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PREFACE 

This  Memorandum  is  a  product  of  a  continuing  study  of  advanced 
techniques  for  the  management  of  research  and  development.  It  presents 
an  exploratory  effort  to  develop  the  framework  of  a  procedure  for  the 
collection  and  analysis  of  data  on  uncertainty  and  progress  regarding 
technical  performance  in  weapon  system  development.  The  data  collected 
are  concerned  with  uncertainty  about  the  characteristics  of  the  compo¬ 
nent  parts  of  the  total  system.  These  data  are  combined,  using  system 
design  relationships,  to  determine  the  uncertainty  about  the  perform¬ 
ance  of  the  total  system.  Application  of  the  procedure  will  involve 
the  use  of  Monte  Carlo  simulation. 

The  procedure  may  be  applied  to  military  or  commercial  development 
projects,  and  it  may  be  used  to  obtain  information  on  uncertainty  at 
the  system,  subsystem,  or  other  levels.  As  such,  it  should  be  of  in¬ 
terest  to  persons  involved  in  the  management  of  research  and  development. 
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SUMMARY 


The  purpose  of  this  Memorandum  is  to  present  a  conceptual  frame¬ 
work  for  measuring  uncertainty  and  progress  regarding  technical  per¬ 
formance  in  a  weapon  system  development  project.  The  emphasis  is  on 
information  of  interest  to  higher-level  military  and  contractor  manage¬ 
ment.  For  this  reason,  the  discussion  is  in  terms  of  characteristics 
such  as  would  appear  in  an  operational  requirement  describing  the  per¬ 
formance  of  the  system. 

The  procedure  is  based  on  four  premises.  First,  progress  in  sys¬ 
tems  development  is  characterized  by  reduction  of  uncertainty;  hence, 
if  uncertainty  can  be  measured  at  different  times,  then  progress  may 
be  indicated  by  changes  in  measures  of  uncertainty.  Second,  assessment 
of  uncertainty  regarding  systems  development  is  subjective;  that  is, 
it  depends  on  individual  beliefs.  Third,  subjective  estimates  of  un¬ 
certainty  can  be  expressed  in  terms  of  probabilities.  Fourth,  the 
amount  of  uncertainty  is  indicated  by  statistical  measures  of  appropri¬ 
ate  probability  distributions. 

Consistent  with  these  premises,  the  procedure  is  concerned  with 
obtaining  the  probability  distributions  for  the  critical  system  per¬ 
formance  characteristics.  The  following  steps  are  involved: 

1.  Find  design  equations  that  relate  subsystem  properties  to 
total  system  performance. 

2.  Obtain  subjective  probabilities  for  subsystem  and  component 
properties  that  influence  total  systems  performance. 

3.  Use  Monte  Carlo  procedures  to  generate  probability  distribu¬ 
tions  for  the  system  performance  characteristics. 

4.  Calculate  statistical  measures  of  system  performance  proba¬ 
bility  distributions. 

5.  Compare  statistical  measures  for  different  time  periods  to 
obtain  indications  of  progress. 
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I. _ INTRODUCTION 


Meaningful  information  on  the  st  tus  and  progress  of  a  weapon  sys¬ 
tem  during  its  development  is  a  critical  input  to  enlightened  military 
decisionmaking.  Major  decisions  regarding  program  acceleration,  con¬ 
tinuation  at  the  present  rate,  change  in  direction,  or  possible  cancel¬ 
lation  depend  heavily  on  careful  evaluation  of  such  information.  How¬ 
ever,  several  years  are  generally  required  before  the  product  of  a 
weapon  development  program  can  be  subjected  to  total  system  testing  and 
the  overall  performance  definitively  evaluated.  Hence,  decisionmaking 
in  the  interim  must  be  accomplished  with  imperfect  inf ormation-- 
imperfect  in  the  sense  that  the  outcome  is  uncertain.  The  best  deci¬ 
sionmaking  in  these  circumstances  must  take  explicit  account  of  the 
uncertainty.  This  can  be  done  by  obtaining  estimates  of  the  various 
possible  values  of  the  future  performance  of  the  weapon  system.  This 
Memorandum  describes  the  framework  of  a  procedure  that  takes  explicit 
account  of  uncertainty,  that  generates  estimates  of  the  various  possible 

values  of  future  performance,  and  that  uses  these  estimates  to  assess 

,  * 
status  and  progress. 

The  major  characteristics  underlying  the  approach  are  the  follow¬ 
ing: 

1.  The  development  of  a  new,  operational  military  system  involves 
combining  a  number  of  subsystem  and  component  technologies--some  new 
or  improved  and  some  current--to  arrive  at  a  new  system  capability.  At 
the  beginning  of  these  projects  two  types  of  information  are  available: 
design  equations  describing  performance  of  the  general  class  of  sys¬ 
tem  in  terms  of  the  characteristics  of  the  components,  and  some  knowl¬ 
edge  about  new  or  improved  technologies  to  be  incorporated  in  certain 
of  the  components  or  subsystems.  On  the  other  hand,  it  is  not  known 
whether  these  technologies  can  be  engineered  and  combined  into  a  sys¬ 
tem  configuration  that  will  deliver  the  desired  new  capability. 

* 

The  relationship  of  this  procedure  to  the  overall  process  of  de¬ 
velopment  is  illustrated  in  Appendix  A. 
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2.  Status  in  development  can  be  indicated  by  a  measure  of  uncer- 

•/( 

tainty,  and  progress  by  change  in  status.  Hence,  progress  is  indicated 
by  reduction  of  uncertainty.  Weapon  system  development  has  bean  de¬ 
scribed  as  the  sequential  purchase  of  information  regarding  some  object 
about  which  the  state  of  knowledge  is  incomplete.  The  technical  sta¬ 
tus  of  a  development  project,  at  any  time,  is  best  indicated  by  the 
accumulated  knowledge  at  that  time.  Progress  is  measured  by  periodic 
assessment  of  the  change  in  the  state  of  know ledge --that  is,  the  in¬ 
formation  obtained — since  the  last  review. 

3.  The  appropriate  variables  for  consideration  in  decisionmaking 
at  the  total  system  level  can  be  expressed  in  a  few  critical  system 
performance  characteristics.  The  significant  characteristics  are  those 
that  indicate  the  capability  of  the  system  to  perform  certain  prescribed 
missions.  For  example,  in  the  case  of  an  aircraft,  the  characteristics 
might  be  speed,  range,  altitude,  etc. 

4.  To  obtain  such  measures  for  the  performance  of  the  total  weapon 
system  requires  a  method  for  systematically  analyzing  the  implications 
of  knowledge  gained  at  the  subsystem  and  component  levels.  Therefore, 
the  appropriate  variables  for  data  collection  are  the  physical  and  per¬ 
formance  characteristics  of  major  subsystems  and  components.  This  is 

so  because,  in  general,  the  most  technically  knowledgeable  people  in¬ 
volved  in  a  system  development  project  are  the  engineers  doing  the  work. 
There  is  no  person  who  can  appraise  the  performance  of  the  system  with 
anything  near  the  same  level  of  confidence  that  an  engineer  can  appraise 
the  performance  of  the  portion  of  the  system  upon  which  he  is  working. 


Measurement  of  status  in  development  has  a  different  basis  than 
measurement  of  status  in  production.  In  production,  status  is  measured 
in  terms  of  how  many  parts  are  manufactured,  assembled,  etc.  The  is¬ 
sue  of  primary  concern  is  when  the  total  job  will  be  done.  In  develop¬ 
ment,  the  main  concern  is  how  the  end  product  will  perform.  Completion 
of  development  is  indicated  by  the  fact  that  the  part's  characteristics 
are  known  with  certainty. 

See  Ref.  17,  pp.  1-3. 

These  characteristics  are  usually  stated  in  a  Required  Opera¬ 
tional  Capability  (ROC)  or  a  Requirements  Action  Directive  (RAD). 
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5.  Because  weapons  development  is  not  a  random  process,  the  use 

of  subjective  probability  distributions  and  quantities  derivable  from 

‘Ark 

them  is  necessary  in  measuring  the  technological  uncertainty  involved. 

The  uncertainty  and  progress  assessment  technique  described  in 
this  Memorandum  is  based  on  the  above  characteristics,  and  it  involves 
the  following  steps: 

1.  Determine  design  equations  that  relate  subsystem  properties 
to  total  system  performance. 

2.  Obtain  subjective  probabilities  for  subsystem  and  component 
properties  that  influence  total  system  performance. 

3.  Use  Monte  Carlo  procedures  to  generate  probability  distribu¬ 
tions  for  the  system  performance  characteristics. 

4.  Calculate  statistical  measures  of  system  performance  proba¬ 
bility  distributions. 

5.  Compare  statistical  measures  for  different  time  periods  to 
obtain  an  indication  of  progress. 


A  subjective  measure  must  be  used  because  it  is  impossible 
to  obtain  an  objective  measure.  This  is  a  consequence  of  the  methodo¬ 
logical  differences  between  measuring  something  that  has  occurred  as 
opposed  to  predicting  something  that  is  going  to  occur.  If  a  measuring 
device  exists  that  can  be  applied  to  an  event  that  has  occurred,  then 
it  can  be  said  that  the  measurement  is  objective,  and  there  is  cer¬ 
tainty  regarding  what  has  happened.  In  the  case  of  predicting  which 
one  of  several  events  will  occur,  the  situation  is  personal,  or  subjec¬ 
tive,  but  the  likelihood  can  be  expressed  quantitatively  in  terms  of 
probabilities.  These  probabilities,  along  with  certain  summary  statis¬ 
tics,  indicate  the  degree  of  uncertainty.  In  some  instances,  probabil¬ 
istic  evaluations  of  the  likelihoods  of  possible  future  events  are 
referred  to  as  being  objective.  Probabilities  are  objective  when  the 
events  are  the  results  of  "random  phenomena." 

The  nature  of  subjective  probabilities  and  techniques  for  obtain¬ 
ing  them  are  described  in  Appendix  B.  Some  of  the  statistical  measures 
that  characterize  probability  distributions  are  reviewed  in  Appendix  D. 
For  further  information  on  random  phenomena,  see  Ref.  24,  p.  2. 

** 

This  type  of  uncertainty  can  be  distinguished  from  several  other 
types;  for  example,  the  uncertainty  regarding  the  enemy  and  his  reac¬ 
tions.  (See  Ref.  14,  pp.  15-16)  In  general,  the  other  types  of  un¬ 
certainty  are  external  to  the  weapons  developer  (contractor)  and  there¬ 
fore  not  directly  relevant  to  the  evaluation  of  contractor  performance. 
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Note  that  the  measures  of  status  and  progress  are  both  related  to 
the  state  of  knowledge  regarding  the  end  product  and  both  are  based  on 
the  same  subjective  inputs  and  design  equations.  The  use  of  design 
equations  and  carefully  determined  subjective  probabilities  will  assist 
in  obtaining  measures  of  uncertainty  and  progress  in  which  caprice  and 
unreasoned  intuition  are  held  to  a  minimum. 
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II.  MEASUREMENT  OF  TECHNOLOGICAL  UNCERTAINTY  (STATUS) 

This  section  deals  with  the  various  levels  and  times  at  which  sub¬ 
jective  probabilities  may  be  obtained  during  a  development  project, 
and  describes  a  process  to  generate  uncertainty  information  about  the 
performance  of  the  total  system,  based  on  similar  information  regarding 
the  subsystems  and  components. 

STRUCTURAL  ELEMENTS  RELEVANT  TO  MEASUREMENT 

The  development  of  a  complex  system  is  accomplished  by  division 
of  effort.  As  work  progresses,  this  division  becomes  more  and  more  de¬ 
tailed.  Hence,  the  framework  or  structure  of  a  weapon  system  develop¬ 
ment  project  can  be  viewed  as  consisting  of  two  major  elements,  namely, 
(1)  decomposition  of  work  into  components  and  (2)  the  time  sequencing 
of  activities. 

Decomposition  into  Work  Elements 

The  division  of  the  total  system  into  its  subsystems,  the  subsys¬ 
tems  into  their  components,  and  so  on  is  frequently  referred  to  as  the 
work  breakdown  structure.  Figure  1  shows  an  example  of  the  first  four 
levels  in  a  work  breakdown  structure  for  an  aircraft  development  proj¬ 
ect.  In  addition,  the  associated  levels  of  management  and  their  areas 
of  responsibility  (performance  characteristics)  are  shown.  For  example, 
the  major  contractor  is  responsible  for  the  performance  of  the  weapon 

system--here  an  aircraf t--and  therefore  for  the  system  performance 
ie 

characteristics . 

The  level  of  weapon  system  performance  may  not  be  the  best  level 
at  which  to  obtain  subjective  probabilities.  It  seems  logical  that 
the  "quality"  of  probability  estimates  regarding  the  possible  outcome 


Not  shown  but  equally  important  is  a  multi-level  structure  of  de¬ 
sign  equations  in  which  the  performance  variables  associated  with  sub¬ 
systems  and  components  are  related  to  the  performance  objectives  for 
the  overall  system.  Ms  feature  will  be  referred  to  again. 
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of  activities  should  be  better  at  low  levels  of  the  management  struc¬ 
ture  than  at  high  levels.  This  is  because  at  high  levels  a  large 
amount  of  time  is  spent  on  managerial  tasks  and  a  small  amount  on  tech¬ 
nical  tasks.  Managerial  decisions  and  actions,  although  perhaps  broader 
in  perspective,  frequently  have  only  indirect  influence  on  technical 


Examples  of  Man axemen t  nml  Examples  of  System, 

Mana^enK'n l  level  Per f ormnnet-  Hierarchy  Subsystem,  and  Component 

Cha  rac ter i s  t lc  s 


Fig,  I  -  An  Example  of  the  Management  and  Performance  Levels  for  an 
Aircraft  Development  Project 


performance.  Detailed  knowledge  concerning  work  in  progress  may  be 
limited.  At  lower  levels  knowledge  of  the  individual  elements  of  work 
is  likely  to  be  more  intimate,  and  the  decisions  and  actions  more  di¬ 
rectly  related  to  the  characteristics  of  the  components  and  subsystems. 
The  most  detailed  knowledge  is  found  at  the  le'  2I  of  the  engineers 
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engaged  in  design  work.  Thus,  considering  the  quality  of  the  input 
data,  the  "best"  approach  is  to  obtain  subjective  probabilities  at  the 
engineering  level  and  use  an  aggregating  process  to  generate  probabil¬ 
ities  at  the  weapon  system  performance  level. 

Time  Sequencing  of  Activities 

The  time  sequence  of  activities  is  important  because  it  relates 
to  the  level  of  detail  to  which  the  work  breakdown  structure  can  be  de¬ 
fined.  At  any  time  before  the  completion  of  a  project,  it  generally 
is  difficult,  if  not  impossible,  to  specify  explicitly  all  of  the  re¬ 
maining  activities  and  the  order  in  which  they  will  be  undertaken. 
Figure  2  illustrates  a  general  sequencing  of  activities  for  a  jet  en¬ 
gine  development  project. 

Specification  of  the  time  sequence  of  activities  is  not  a  critical 
element  of  this  study.  The  important  point  is  that  work  progresses  to 
finer  levels  of  detail  during  the  course  of  the  project.  This  is  il¬ 
lustrated  in  Fig.  3.  The  development  of  a  new  operational  weapon  sys¬ 
tem  begins  with  an  evaluation  of  needs  and  opportunities  that  leads  to 
delineation  of  desired  characteristics  for  the  total  system.  Prior  to 
the  proposal  phase,  the  subsystems  and  components  usually  are  not  ana¬ 
lyzed  in  detail.  During  the  proposal  phase,  the  gross  characteristics 
and  key  details  of  the  components  are  analyzed.  After  the  contract  is 
granted,  the  analysis  proceeds  down  to  the  level  of  individual  compo¬ 
nents  and  parts.  By  that  time,  sufficient  engineering  analyses  will 
have  been  made  regarding  state-of-the-art  advances  and  off-the-shelf 
items  to  justify  obtaining  subjective  probabilities  at  these  detailed 
levels.  The  measurement  scheme  described  in  this  Memorandum  is  de¬ 
signed  to  monitor  changes  in  uncertainty  regarding  the  system's  per¬ 
formance,  based  on  information  regaraj.ng  the  components  and  subsystems. 

AGGREGATION  TECHNIQUES 

Having  argued  that  "good"  probabilistic  estimates  can  be  obtained 
for  the  characteristics  of  the  components  of  a  total  weapon  system  that 
is  being  developed,  it  is  necessary  to  demonstrate  how  these  estimates 
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General  Design  Study 

Development 

|  t  u*n  ton  Spec  l  f  led 

( Refers  to  component 
innovations  not  yet 
incorporated  in  an 
eng  ine . ) 

1 

( 1)  Spec l f lent  Ion  of 
the  basic  principles 
of  construction. 
f2>  Tentative  spec  if  l* 

1 

Demonstrat Ion  of 
performance  capability 
by  either: 

(1)  Attempting  to 

cation  of  dimensions 
and  shapes  of  major 
components . 

O)  Derivation  of 
performance  curves. 

build  and  run  a 
Performance -or  ten  ted 
Demonstrator  Engine, 
or 

( 2 )  Constructing  and 
testing  components 

impact  on  engine 
performance  but  for 
which  no  test  data 
exist . 

Uelght-and 

Du  gab H ity-or lentcd 

Development 


(1)  Selection  of 
light  weight  and 
durable  materials. 

( 2)  Working  out 
minute  configuration 
details  of  major 
components , 


Final  Prototype 


(1)  Construction  of 
complete  operational 
eng  ine. 

(2)  Testing. 


Fig.  2  -  A  Generalized  Sequence  of  Activities  in  Jet  Engine  Development  (Adopted  from  Ref.  22) 


Fig.  3  -  An  Illustration  of  Increasing  Detail  of  Design 


BEST  AVAILABLE  COPY 
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can  be  analyzed  and  manipulated  to  arrive  at  information  about  the  per¬ 
formance  of  the  total  system.  As  was  referenced  above,  the  performance 
of  a  weapon  system  can  be  expressed  in  terms  of  the  characteristics  of 
its  components  by  means  of  design  equations.  Some  examples  are  given 
in  the  next  section. 

Given  the  design  equations  and  probability  distributions  for  the 
component  characteristics  that  appear  in  the  design  equations,  then  it 
is  possible  to  generate  probability  distributions  for  the  system  per¬ 
formance  characteristics.  This  can  be  done  analytically;  however, 
mathematical  analysis  is  feasible  only  when  the  probability  distribu¬ 
tion  are  simple  cr  of  standard  form  and  the  design  equations  satisfy 

•k 

certain  restrictions.  Because  these  conditions  will  almost  never  be 
satisfied,  analytical  techniques  appear  inappropriate.  Instead, 
simulation  must  be  used. 

The  method  described  here  uses  Monte  Carlo  simulation,  in  which 
the  subjective  probability  distributions  for  the  component  character¬ 
istics  are  subjected  to  random  sampling.  The  values  obtained  are  then 
used  to  calculate  values  for  the  system  performance  characteristics. 
This  process  is  repeated  a  large  number  of  times  until  distributions 

over  the  possible  values  of  the  system  performance  characteristics  are 
** 

obtained.  In  this  way,  Monte  Carlo  simulation  is  used  to  generate 
probability  distributions  for  the  critical  system  performance  charac¬ 
teristics  (see  Fig.  4). 

The  following  example  demonstrates  the  nature  of  the  design  equa¬ 
tions,  the  probability  distributions  for  the  component  characteristics, 
and  the  probability  distributions  for  the  system  performance  character¬ 
istics.  Discussion  of  techniques  to  explicitly  account  for  various 
types  of  interdependence  follows  the  example. 


For  a  discussion  of  the  use  of  analytical  techniques  with  normal 
probability  distributions  see  Ref.  1.  Of  course,  non-normal  distribu¬ 
tions  can  be  modified,  or  the  engineers  can  be  questioned  in  such  a  way 
that  their  responses  are  "normal."  However,  this  is  believed  t:o  intro¬ 
duce  unnecessary  "inaccuracy."  See  Appendix  B. 

** 

Among  the  numerous  ways  to  determine  the  exact  number  of  times 
that  the  process  should  be  repeated  are:  (1)  to  assume  that  the  re¬ 
sultant  distributions  are  normal  and  use  the  appropriate  statistical 
rules,  and  (2)  to  obtain  distributions  of  large  numbers  of  small  sam¬ 
ples  and  thereby  take  advantage  of  the  law  of  large  numbers.  These 
two  courses  of  action  can  be  evaluated  using  statistical  techniques 
found  in  standard  texts  such  as  Ref.  23. 
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Fig.  4  -  The  Inputs  and  Outputs  of  the  Aggregation  Process 
AN  EXAMPLE 


Step  1.  Obtain  Design  Equations 

A  design  equation  is  a  functional  relationship  between  a  system 
performance  characteristic  and  the  subsystem  and  component  character¬ 
istic  that  determine  the  system's  performance.  Such  equations  are  de¬ 
rived  from  empirical  data  and  they  are  approximate  representations  over 
the  total  range  of  the  data.  Consequently,  in  many  cases  more  realis¬ 
tic  results  can  be  obtained  by  using  the  empirical  data.  The  discus¬ 
sion  that  follows  uses  design  equations  exclusively.  However,  it  should 
be  noted  that  the  technique  works  regardless  of  the  form  of  the  data. 

The  design  equations  are  merely  a  representation  of  the  data  that  ..llows 
the  user  to  calculate  values  of  the  system  performances  for  any  combi¬ 
nation  of  component  characteristics.  A  similar  result  could  be  obtained 
by  reading  the  empirical  data  in  tabular  or  graphical  form.  In  a  real 
application,  both  design  equations  and  graphical  (or  tabular)  repre¬ 
sentations  would  probably  be  used. 

Because  of  the  approximate  nature  of  design  equations  and  the  de¬ 
sire  of  engineers  and  others  to  avoid  certain  types  of  inadequate  de¬ 
sign,  many  of  the  equations  have  been  modified  to  include  safety,  or 
"fudge"  factors.  For  example,  a  constant  may  be  added,  or  subtracted, 
to  provide  a  built-in  margin  of  safety.  These  modified  equations 
give  a  less  accurate  representation  of  reality  than  their  unmodified 
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forms;  hence,  they  are  not  as  appropriate  for  predicting  the  character¬ 
istics  of  the  total  system  based  on  the  characteristics  of  the  parts. 

The  design  equations  used  in  the  measurement  technique  should  be 
the  equations  used  by  the  engineers  who  are  doing  the  design  work. 
However,  when  the  equations  are  used  for  predicting  system  performance, 
any  known  "fudge"  factors  should  be  removed. 

The  amount  of  detail  in  the  design  equations  will  depend  on  how 
much  the  various  states  of  the  art  are  being  advanced.  For  example, 
if  an  existing  engine  is  to  be  used,  then  only  the  gross  characteris¬ 
tics  of  the  engine  that  influence  system  performance  (for  example, 
thrust,  weight,  fuel  consumption,  etc.)  need  be  considered.  If  a  new 
engine  is  being  developed,  then  further  details  of  the  engine  must  be 
considered  in  order  to  determine  the  gross  characteristics;  that  is,  a 

irk 

design  equation  for  the  engine  is  necessary.  Thus,  there  is  a  hier¬ 
archy  of  design  equations,  and  the  level  of  detail  required  for  assess¬ 
ing  status  depends  on  the  degree  of  technological  advancement  being 

kkk 

incorporated.  The  design  equations  in  this  summary  example  are  for 


If  there  are  two  or  more  design  equations  and  there  is  no  agree¬ 
ment  about  which  one  is  correct,  then  it  may  be  useful  to  make  a  prob¬ 
abilistic  combination  of  the  competing  design  equations. 
kk 

There  may  be  cases  in  which  a  choice  has  not  been  made  between 
using  an  existing  component  and  developing  a  new  one.  Methods  for 
dealing  with  such  cases  are  discussed  at  the  end  of  this  section. 

kick 

At  this  point  it  may  be  useful  to  summarize  the  various  obser¬ 
vations  regarding  level  of  detail.  It  does  not  seem  likely  that  hard 
and  fast  rules  will  evolve;  rather  the  objective  can  be  stated  gener¬ 
ally  as  the  minimum  detail  consistent  with  obtaining  the  ability  to 
realistically  sense  (measure)  status.  As  mentioned  above,  the  technique 
should  be  applied  in  more  detail--more  levels  of  design  equations-- 
on  subsystems  and  components  involving  new  technology,  that  is,  where 
advances  in  state  of  the  art  introduce  greater  uncertainty.  Second, 
the  technique  should  be  carried  to  the  level  where  knowledgeable  engi¬ 
neers  are  most  closely  involved  with  those  technical  aspects  of  the  de¬ 
sign  which  bear  directly  on  ultimate  performance  of  the  weapon  system. 
Third,  as  will  be  discussed  subsequently,  it  may  be  desirable  to  apply 
the  process  to  an  additional  level(s)  to  achieve  statistical  indepen¬ 
dence  in  the  basic  subjective  probability  inputs.  These  considerations 
must  be  balanced  against  the  objective  of  avoiding  a  technique  that  is 
excessively  burdensome  to  either  the  engineers  providing  the  estimates 
or  the  administrative  staff  processing  them  and  preparing  management 
reports . 
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i.  1  lustration  only. 

Consider  a  project  to  develop  an  aircraft  and  suppose  that  the  per 

formance  characteristics  that  are  critical  to  the  aircraft's  mission 
capability  are  speed,  altitude,  range .  and  endurance .  The  design 
equations  determine  the  component  characteristics  for  which  subjective 
probabilities  are  required. 

Speed.  The  design  equation  for  the  maximum,  constant  altitude, 
level-flight  speed  of  an  aircraft  is  given  by 


V 

max 


195.5 


T  + 
max 


1.274 


(1) 


where  T  is  the  maximum  thrust  available  (assumed  independent  of 
max 

speed),  is  the  drag  coefficient  for  zero  lift,  S  is  the  wing  area, 

W  is  the  gross  weight  of  the  aircraft,  a  is  the  altitude  density  ra- 

tio,  e  is  the  efficiency  factor,  and  b  is  the  wing  span. 

Altitude .  One  way  to  determine  an  aircraft's  altitude  capability p 

H  ,  is  to  determine  the  ambient  pressure  ratio  at  the  absolute  ceil- 
max 

ing  and  then  use  a  table  of  the  ambient  pressure  ratio  as  a  function 

of  altitude  to  determine  the  altitude.  One  form  of  the  equation  for 

*** 

the  ambient  pressure  ratio  at  the  absolute  ceiling  is 


P 

ac 


/cdV  (Wp/T) 

VnAe/  cr(W  /W  ) 
'  '  P  8 


(2) 


* 

Obtained  from  Standard  Atmosphere  Table,  Ref.  25,  pp.  481-482. 

"frit 

Obtained  by  solving  Eq .  (4-2),  Ref.  25,  for  V.  This  formulation 
assumes  a  parabolic  variation  of  Cq  versus  and  does  not  Include  con¬ 
sideration  of  compressibility  speed  corrections.  It  is  used  when  the 
speed  is  limited  by  the  thrust  available. 

irk* 

Ref.  4,  p.  1.  Again,  this  equation  is  for  the  thrust-limited 
celling.  If  the  service  ceiling  is  the  important  characteristic,  then 
the  design  equation  for  that  quantity  should  be  used. 
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Using  A  =  b  /S, 


this  can  be  written  as 


P 

ac 


(3) 


where  S,  T,  W  ,  b,  and  e  are  defined  above;  W  is  the  weight  of  the 
8  P 

power  plant;  and  a  is  the  thrust  lapse  rate  factor. 

Range .  The  maximum  range  (in  miles)  of  a  turbojet  aircraft  at  a 

* 

constant  altitude  is  given  by  the  following  equation: 


(4) 


'D  /max 


(5) 


2 

Substituting  Eq.  (5)  into  Eq.  (4),  and  again  using  A  =  b  /S  gives 


where  C1  is  the  fuel  consumption  measured  in  pounds  per  hour  per  pound 
of  thrust,  and  is  the  gross  weight  of  the  aircraft  at  the  end  of 
its  mission. 

Endurance .  The  maximum  endurance  (in  hours)  of  a  turbojet  air¬ 
craft,  independent  of  altitude,  is  given  by 


Equation  (4)  gives  the  maximum  "  ...  range  within  3  percent  of 
the  absolute  optimum  for  most  airplane-engine  combinations."  (Ref.  25, 
pp.  191-192.) 

Ref.  25,  p.  191. 
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Again,  for  parabolic  polar  variation  of  versus  C. 

U  Li 


The  system  performance  characteristics  and  component  characteris¬ 
tics  are  summarized  in  Table  1.  The  inputs  for  the  simulation  in  this 
example  are  the  design  equations  (Eqs .  (1),  (3),  (6),  and  (9)),  and  the 
probability  distributions  for  the  component  characteristics. 

Step.  2.  Determine  Subjective  Probabilities 

Some  sample  probability  distributions  for  the  component  character¬ 
istics  are  shown  in  Table  2.  Techniques  for  determination  and  solici¬ 
tation  of  probability  estimates  are  discussed  in  Appendix  B. 

In  this  table  it  has  been  assumed  that  if  the  initial  gross  weight 
of  the  aircraft  is  known,  the  final  gross  weight  is  determined  uniquely. 

For  example,  if  W  -  250,000  lb,  then  W,  *  130,000  lb.  The  purpose  of 
8 

this  assumption  is  simplification.  In  general,  for  a  given  W^,  will 
usually  vary  over  some  range;  that  is,  for  each  possible  value  of  W  , 
there  will  be  a  probability  distribution  for  W^.  The  same  relation¬ 
ship  is  assumed  for  b  and  S. 

^Initial  and  final  weights  refer  to  the  weight  of  an  aircraft  at 
the  beginning  and  end  of  its  mission. 

See  section  on  "Correlation  Among  Inputs,"  p.  20. 


Tablo  1 

PERFORMANCE  AND  COMPONENT  CHARACTERISTICS  FOR  HYPOTHETICAL 
AIRCRAFT  DEVELOPMENT  PROJECT 


Performance  Character  1st. i  cs 


Component 

Character¬ 

istics 

Units 

Speed  (V  ) 

r  v  max' 

mi/hr 

Altitude'1  (H  ) 

max 

ft 

Range  (R  ) 
max 

mi 

Endurance  (E  ) 

max 

hr 

W 

8 

lb 

X 

X 

X 

X 

Wf 

lb 

X 

X 

C 

Q 

O 

X 

b 

ft 

X 

X 

X 

X 

S 

ft2 

X 

X 

X 

X 

e 

X 

X 

X 

X 

T 

lb 

X 

X 

a 

X 

C' 

lb/hr/ lb 
of  T 

X 

X 

aRecall  that  the  design  equation  gives  the  ambient  pressure  ratio  at  absolute 
ceiling.  The  corresponding  altitude  is  found  in  a  fable  of  ambient  pressure  ratio 
versus  altitude. 
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Tub  lo  2 

PROBABILITY  DISTRIBUTIONS  FOR  COMPONENT  CHARACTERISTICS  (INPUTS) 
IN  HYPOTHETICAL  AIRCRAFT  DEVELOPMENT  PROJECT 


W  (lb) 

250,000 

260,000 

270,000 

280,000 

290,000 

Wf  (lb) 

130,000 

140,000 

150,000 

160,000 

170,000 

P(W) 

.05 

.10 

.35 

.30 

.20 

CD 

0 

.015 

.016 

.017 

.018 

-- 

p(cD  ) 

c, 

.10 

.40 

.30 

.20 

-- 

b  (ft) 

170 

175 

180 

185 

190 

s  (ft2) 

3,700 

3,800 

3,900 

P(S) 

.15 

|  .15 

.35 

.25 

.10 

e 

.70 

.75 

.80 

.85 

.90 

P(e) 

.20 

.25 

.35 

.15 

.05 

T  (lb) 

65,000 

70,000 

75,000 

80,000 

-- 

P(T) 

.15 

.35 

.25 

.25 

-- 

a 

1.2 

1.6 

1.8 

-- 

p(°0 

.20 

.25 

.30 

.25 

-- 

C ' ( lb/hr/ lb 
of  T) 

.80 

.85 

.90 

.95 

_  _ 

P(C') 

.30 

.30 

.25 

.15 

-- 

Step  3.  Generate  Probability  Distributions 
for  System  Performance 

The  outputs  from  the  simulation  are  the  probability  distributions 
for  the  aircraft  performance  characteristics.  Samples  of  these  distri 
butions  are  shown  in  Table  3.  These  figures  are  merely  representative 
of  the  type  and  form  of  the  output  distributions;  they  were  not  ob¬ 
tained  by  carrying  out  the  simulation  using  the  design  equations  and 
the  information  in  Table  2.  The  process  of  generating  probability 
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Table  3 

PROBABILITY  DISTRIBUTIONS  FOR  SYSTEM  PERFORMANCE  CHARACTERISTICS 
(OUTPUTS)  IN  HYPOTHETICAL  AX'  .1AFT  DEVELOPMENT  PROJECT 


V 

max 

(mph) 

650 

675 

700 

725 

750 

P(V) 

.15 

.20 

.45 

.  10 

.  10 

H 

max 

(ft) 

45,000 

50,000 

55,000 

60,000 

65,000 

P(H) 

.10 

.  20 

.40 

.  25 

R 

max 

(mi) 

3,750 

4,000 

4,250 

4,500 

4,750 

P(R) 

.25 

.35 

.20 

.10 

.10 

E 

max 

(hr) 

13.50 

13.75 

14.00 

14.  25 

14.50 

P(E) 

.10 

.15 

.15 

.40 

.20 

distributions  using  Monte  Carlo  simulation  is  discussed  in  Appendix  C.  The 

discussion  in  Appendix  C  uses  a  hand-calculated  example.  In  any  real 

application  of  the  procedure  described  in  this  Memorandum,  a  computer- 

* 

ized  simulation  would  be  required  because  of  the  size  of  the  problem. 

Responsibility  for  performing  the  Monte  Carlo  simulation  has  some 
bearing  on  what  items  of  information  are  reported  to  various  management 
levels.  Consider  the  two  highest  levels  shown  in  Fig.  1.  If  the  gov¬ 
ernment  agency  performs  the  simulation,  it  receives  the  design  equations 
and  the  probability  distributions  for  the  variables  in  the  design  equa¬ 
tions  from  the  major  contractor.  If  the  major  contractor  performs  the 
simulation,  the  information  reported  to  the  government  agency  is  the 
output  of  the  simulation.  In  the  latter  case,  the  government  agency 
may  also  want  the  input  data  for  the  simulation  as  an  indication  of  the 
status  of  the  subsystems  and  components. 


An  estimate  of  the  computer  time  required  for  the  aircraft  devel¬ 
opment  example  is  obtained  as  follows:  The  equivalent  of  approximately 
200  multiplications  is  required  to  obtain  one  sample  from  each  of  the 
probability  distributions  in  Table  2,  and  to  calculate  the  values  of 
the  system  performance  characteristics.  Thus  20,000  multiplications 
would  be  required  for  a  sample  of  100.  On  a  computer  with  a  multiply 
time  of  ten  microseconds  (10-^  sec)  the  total  time  would  be  one-fifth 
of  a  second. 


Step  4.  Ca lculate  Statistical  Measures  of  Status 

The  probability  distributions  for  the  critical  system  performance 
characteristics  represent  the  state  of  knowledge  regarding  the  system 
being  developed.  However,  the  data  in  this  form  do  not  give  an  unam¬ 
biguous  picture  of  status.  The  problem  is  interpretation  of  probabil¬ 
ity  distributions--what  does  one  distribution  signify  and  how  does  it 
compare  with  another? 

Probability  distributions  are  characterized  by  certain  statistical 
measures- -measures  of  central  tendency,  measures  of  dispersion,  etc. 
Measures  of  central  tendency,  such  as  the  mean,  median,  and  mode,  give 
an  indication  of  where  the  center  of  the  distribution  is  located,  or 
in  the  current  context,  a  single  estimate  of  what  performance  level  is 
likely  to  be  attained.  Measures  of  dispersion,  such  as  the  range,  vari¬ 
ance,  and  standard  deviation,  give  an  indication  of  how  widely  the  dis¬ 
tribution  is  spread,  or  how  much  variation  about  the  expected  perform¬ 
ance  is  likely.  As  an  example,  for  the  speed  distribution  in  Table  3, 
the  mean  is  695  mph,  and  the  standard  deviation  is  28  mph. 

When  required  values  for  the  performance  characteristics  are  spec¬ 
ified,  the  probability  of  meeting  or  exceeding  the  specification  is  of 
particular  interest.  If  the  speed  specification  in  the  example  above 
is  700  mph,  then  (from  Table  3)  the  probability  of  meeting  or  exceed¬ 
ing  the  specification  is  .65.  Each  of  the  statistical  measures  used 
in  this  Memorandum  is  reviewed  and  placed  in  the  context  of  performance 
measurement  in  Appendix  D.  Their  usage  in  tracking  changes  in  status 
over  time  (measuring  progress--Step  5),  and  their  sensitivity  to  changes 
in  time  and  cost  are  discussed  in  Sec.  III.  The  remainder  of  this  sec¬ 
tion  deals  with  techniques  foi  handling  various  sources  of  interde¬ 
pendence  . 

DEALING  WITH  INTERDEPENDENCIES 

Probability  of  Meeting  or  Exceeding  All  Specifications: 

Correlation  Among  Outputs 

In  the  aircraft  development  example,  separate  probability  distri¬ 
butions  are  generated  for  each  system  performance  characteristic.  This 
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process  treats  the  performance  characteristics  as  being  independent. 

When  this  is  done,  statements  can  be  made  only  about  meeting  or  exceed¬ 
ing  each  system  performance  characteristic  independently  of  all  the 
others.  For  example,  consider  the  data  in  Table  3  and  suppose  that  the 
specifications  are  =  675  mph,  =  50,000  ft,  =  4000  mi,  and  E 
=  13.75  hr.  Then  the  probability  of  meeting  or  exceeding  the  speed 
specification  is  p(V  >  V  )  =  .85,  independent  of  the  values  of  the  other 
performance  characteristics.  Similarly,  p(H  -  H^)  =  .90;  p(R  ^  R^)  = 
.75;  and  p(E  s  E  )  =  .90,  all  independent  of  each  other. 

If  the  performance  characteristics  above  were  independent,  then 
the  (joint)  probability  of  meeting  or  exceeding  all  of  the  specifica¬ 
tions  is  given  by  the  product  of  the  separate  probabilities  of  (inde¬ 
pendently)  meeting  or  exceeding  the  specifications: 

p(V  £  V  ,  H  £  Hr,  R  a  Rf,  E  Sr  Ef)  = 

P(V  ^  Vr)p(H  *  Hr)p(R  5  Rr)p(E  s  Er)  =  .516 

However,  the  performance  characteristics  above  are  not  independent  and 
it  is  not  valid  to  make  a  joint  statement  based  on  independently  gen¬ 
erated  probability  distributions.  It  _is  valid  to  make  independent 
statements  concerning  the  individual  characteristics,  and  if  the  inde¬ 
pendent  statements  are  satisfactory,  then  it  is  not  necessary  to  use 
the  slightly  more  complicated  procedure  described  in  the  next  two 
paragraphs . 

When  two,  or  more,  system  performance  characteristics  have  one, 
or  more,  common  component  characteristic,  the  system  performance  char¬ 
acteristics  are  not  independent.  In  the  example,  all  system  perform¬ 
ance  characteristics  depend  on  the  wing  span  and  wing  area;  hence,  they 
are  correlated.  When  system  performance  characteristics  are  correlated, 

then  the  joint,  or  multivariate,  probability  distribution  over  all  per- 

* 

formance  characteristics  must  be  generated.  This  is  done  by  sampling 
all  the  component  probability  distributions  to  obtain  one  value  for 

See  Appendix  B  for  a  brief  description  of  multivariate  distribu¬ 


tions 
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each  characteristic.  Then  values  for  all  of  the  system  performance 

VC 

characteristics  are  calculated.  These  values  describe  one  possible 
performance  "configuration,"  or  vector,  for  the  system. 

Using  the  previous  example,  the  probability  distributions  shown 
in  Table  2  are  sampled  to  obtain  one  value  for  each  component  charac¬ 
teristic.  Then,  values  for  all  four  system  performance  characteristics 
are  calculated  using  the  design  equations  (Eqs.  (1),  (3),  (6),  and  (9)). 
These  values  are  recorded  as  a  four-dimensional  vector.  A  typical  vec¬ 
tor  might  be  (V  =  700  mph,  H  =  50,000  ft,  R  =  4000  mi,E  =  14.00  hr). 

The  process  is  repeated  until  a  "sufficiently  large"  number  of  vectors 
has  been  generated.  The  probability  that  the  aircraft  will  meet  or 
exceed  all  o_  the  specifications  is  the  ratio  of  the  number  of  vectors 

for  which  V  ■  V  ,  H  >  H  ,  R  -  R  ,  and  E  >  E  to  the  total  number  of 
_ r_ _ rj _ r - - r _ 

vectors .*** 


Correlation  Among  Inputs 

In  addition  to  correlation  among  the  outputs,  there  may  be  corre¬ 
lation  among  the  input  quantities  (those  quantities  for  which  subjec¬ 
tive  probabilities  are  obtained  from  the  engineers).  This  will  neces¬ 
sarily  complicate  the  process  of  determining  the  subjective  probabili¬ 
ties.  In  some  instances  it  may  be  possible  to  eliminate  the  correlation. 
Frequently  component  performance  characteristics  that  appear  to  be  cor¬ 
related  are  composed  of  smaller  components  whose  characteristics  are 
independent.  Because  it  is  easier  to  handle  probability  estimates  for 
Independent  characteristics,  it  may  be  worthwhile  to  further  decompose 
a  subsystem  and  thereby  achieve  independence  at  the  input  level. 


*The  same  design  equations  and  component  probability  distributions 
are  used. 

In  general,  a  larger  sample  is  required  to  obtain  this  type  of 
output  at  the  same  level  of  confidence  as  in  the  independent  case.  (See 
footnote  p.  9  and  Ref.  23.) 

*’f*Ca leu lat ions  involving  multivariate  distributions  become  very 
involved  as  the  number  of  variates  increases.  This  is  due  to  an  in¬ 
crease  in  the  number  of  arithmetical  operations  required.  In  addition, 
there  is  no  single  measure  of  variance.  Instead,  there  is  a  covariance 
between  each  pair  of  variates.  (See  Ref.  23,  pp.  102-103,  for  a  brief 
discussion. ) 
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In  other  instances,  however,  this  may  not  be  the  preferred  ap¬ 
proach.  As  an  example,  suppose  that  at  some  time  before  wind  tunnel 
tests  have  been  concluded  it  is  desired  to  obtain  probability  distri¬ 
butions  for  the  span  and  area  of  the  wing.  If  the  geometrical  shape 
of  the  wing  is  specified,  then  the  span  determines  the  area,  and  vice 
versa.  If  the  shape  is  net  specified,  then  the  span  does  not  determine 
the  area.  However,  there  is  usually  a  limited  range  of  possible  shapes 
and  the  amount  that  the  area  varies  as  the  span  changes  is  limited  by 
the  possible  shapes.  Thus,  the  span  and  area  are  correlated. 

There  are  three  feasible  courses  of  action  in  such  a  situation. 

One  is  to  ignore  the  correlation  and  obtain  independent  distributions 
for  the  characteristics.  This  is  justified  when  the  correlation  is 
very  weak--al.most  no  correlation  at  all.  When  the  correlation  is  very 
strong,  complete  correlation  can  be  assumed  and  one  distribution  ob¬ 
tained,  as  shown  in  the  case  of  initial  and  final  gross  weight  in  the 
preceding  example.  The  third  option  is  to  take  explicit  account  of 
the  correlation  by  obtaining  multivariate  probability  distributions. 

The  engineer  is  best  qualified  to  select  the  course  of  action  to  be 
taken,  based  on  which  one  produces  data  that  best  represent  his  evalu¬ 
ation  of  the  degree  of  interdependence.  In  making  his  decision  the 
engineer  should  recognize  that  the  use  of  multivariate  distributions 
for  component  characteristics  introduces  some  complications  in  both 
solicitation  techniques  and  sampling  procedures.  Procedures  based  on 
multi-stage  application  of  techniques  used  with  noncorrelated  variables 
can  be  constructed  for  handling  the  sampling  complication.  These  pro¬ 
cedures  are  straightforward  and  therefore  not  discussed  in  this  Memoran¬ 
dum.  Both  of  these  complications  entail  increased  costs  and  some  anal¬ 
ysis  should  be  made  to  compare  the  benefits  of  more  realistic  inputs 
to  the  costs  of  obtaining  and  using  them. 

UNCERTAINTY  ARISING  FROM  COMPETING  DESIGNS 

The  preceding  discussion  has  implicitly  assumed  that  all  choices 
between  alternative  subsystem  and  component  designs  have  been  made. 

When  any  of  the  choices  remain  undecided  they  contribute  to  the  over¬ 
all  uncertainty  and  their  effect  must  be  included.  In  many  system 


-22- 


development  programs  these  choices  are  purposely  postponed  until  a 
configuration  with  a  high  probability  of  success  is  discovered. 

In  a  given  system  development  program,  there  may  be  several  com¬ 
ponents  for  which  there  are  competing  designs.  From  this  multiplicity 
of  undecided  design  choices  it  can  be  seen  that  there  are  many  possible 

combinations  of  component  designs,  each  of  which  is  a  possible  final 

configuration  for  the  total  system.  The  representation  of  the  uncer¬ 
tainty  with  regard  to  choice  depends  on  how  the  choices  are  made.  The 
following  two  cases  can  be  distinguished:  (a)  The  choice  of  a  design 
for  each  component  may  be  made  without  consideration  of  the  various 
combinations  of  designs  for  other  components;  in  this  case  the  choice 

is  independent ;  and  (b)  The  choice  may  be  made  on  the  basis  of  the 

characteristics  of  the  various  combinations--that  is,  the  characteris¬ 
tics  of  the  various  system  configurations;  in  this  case  the  choice 
is  correlated. 

The  following  two  schemes  will  yield  a  single  set  of  probability 
distributions  for  the  system  performance:  (1)  For  each  component,  com¬ 
bine  the  sets  of  probability  distributions  for  the  competing  designs 
into  one  set.  This  will  yield  a  set  of  probability  distributions  for 
an  "average"  component.  These  distributions  and  the  design  equations 
are  used  to  generate  a  single  set  of  probability  distributions  for  the 
system.  (2)  For  each  combination  of  competing  component  designs-- that 
is,  for  each  configuration  of  the  system- -generate  a  set  of  probability 
distributions  for  the  system  performance.  There  will  be  as  many  sets 
of  system  performance  distributions  as  there  are  combinations  of  com¬ 
peting  component  designs.  These  sets  of  distributions  are  then  com¬ 
bined  into  one  set  for  the  system.  The  mechanics  of  effecting  the  com¬ 
binations  for  both  schemes  can  be  illustrated  using  the  following  example: 

Consider  a  project  to  develop  a  system  that  has  two  compo¬ 
nents  A  and  B.  Suppose  that  each  component  has  two  competing  de¬ 
signs,  and  A^  ,  and  and  B^  and  let  ^A2  ’  ^Bl*  ^*B2 

represent  their  probability  distributions. 

Under  the  first  scheme  the  probability  distributions  for  the 
"average"  components  are  found  by  carrying  out  the  following 


combinations : 
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P_  =  p(Ai)Pa1  +  P(A2)PA2  and 
A 


p-  ■  p<BX>pB1 
B 


P<'B2^PB2  : 


where  p(A^)  is  the  probability  that  design  A.  is  chosen,  p(A2)  is 

* 

the  probability  that  design  A„  is  chosen,  and  p_  and  p  rep- 

2  A  B 

resent  the  probability  distributions  for  the  "average"  components. 
Suppose  two  designs  for  a  jet  engine  are  being  pursued  and  the 
probability  distributions  for  the  thrust  of  the  engine  for  each 
design  are: 


Design  Aj,  p^T  =  17,000  lb)  =  0.6,  p^T  =  18,000  lb)  =  0.4; 

Design  Aj ,  p2 (T  =  17,000  lb)  =  0.3,  P2 (T  =  18,000  lb)  =  0.7. 

If  the  probability  that  Design  A^  will  be  selected  is  p(A^)  =  0.4 
and  the  probability  that  Design  will  be  selected  is  pCA^  =  0.6, 
then  the  probability  distribution  for  the  thrust  of  the  "average" 
component  or  subsystem  (jet  engine)  is  given  by 

p(T  =  17,000  lb)  =  p1(T  =  17,000  lb)  x  pCAj)  +  p£ (T  =  17,000  lb)x  p(A  ) 
-  0.42,  and  P(T  =  18,000  lb)  =  p1(T  «  18,000  lb) 
x  P(Aj^)  +  p2(T  =  18,000  lb)  x  p(A2)  *  0.58. 


Probability  distributions  for  the  system  performance  are  obtained 
by  applying  the  Monte  Carlo  procedure  using  the  design  equations 
and  the  probability  distributions  for  the  "average"  components. 

The  second  scheme  begins  by  using  Monte  Carlo  to  generate  a 
set  of  probability  distributions  for  the  system  performance  for 
each  combination  of  the  various  component  designs.  One  set  of 
distributions  for  the  system  is  obtained  by  combining  the  sets  of 
probability  distributions  corresponding  to  the  various  combinations 


Note  that  probabilities  p(A  )  and  pCA^  are  independent  of  proba¬ 
bilities  p(B. )  and  p(B2).  This  reflects  the  independent  choice  condi¬ 
tions  described  above. 
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of  component  designs--that  is,  the  various  conf igurations--using 
the  probabilities  that  the  various  combinations  (configurations) 
are  chosen.  In  terms  of  the  example,  there  are  four  combinations 
of  competing  designs  (A.,  B^,  (Aj,  B2),  (A^  and  (A2 ,  B2>. 

The  Monte  Carlo  procedure  yields  four  sets  of  probability  distri- 
butions  for  the  system  performance,  one  for  each  combination  of 
competing  designs.  Denote  these  by  pgH,  pgl2>  pg21>  and  pg22. 

The  probabilities  that  the  four  design  combinations  will  be 
chosen  can  be  represented  as  p(Alf  B^ ) ,  p(AL,  B  ),  p(A2>  B1>,  and 
P(A2>  B2).  The  single  set  of  probability  distributions  for  the 
system  is  then  found  by  carrying  out  the  following  combination: 

ps  -  p(.y  »,)psu  +  P<\.  b2)ps12  +  P(A2,  b1)ps21  +  p(A2,  b2)ps22, 

where  p  is  the  single  set  of  probability  distributions  for  the 
system. 

The  following  points  snould  be  noted  with  regard  to  the  two 
schemes : 

(a)  The  first  scheme  can  only  be  used  under  independent 
choice  conditions.  In  terms  of  the  example,  the  probability  that 
either  of  the  designs  A^  or  A2  is  chosen  does  not  depend  on  the 
probability  that  either  of  the  designs  or  B0  is  chosen.  When 
the  purpose  is  to  arrive  at  one  set  of  probability  distributions 
for  the  system,  the  first  scheme  is  superior  because  of  the  shorter 
time  required  to  perform  a  single  Monte  Carlo  as  compared  to  the 
many  Monte  Carlos--one  for  each  possible  design  combination- -with 
the  second  scheme.  However,  if  there  are  correlated  design  choices, 
then  the  second  scheme  must  be  used. 

(b)  The  second  scheme  can  be  used  with  both  independent 
choice  and  correlated  choice.  This  is  because  there  are  two  ways 
to  obtain  the  probabilities  that  the  various  design  combinations 
are  chosen.  One  way  is  to  use  the  independently  obtained 

'ft 

As  will  be  noted  later  in  this  section,  there  are  two  ways 
to  obtain  these  probabilities. 


-25- 


probabilities  that  were  used  to  determine  the  average  components. 

In  the  terms  of  the  example:  p(Aj,  B^)  =  pCA^pCBj),  pCA^  B2) 

=  P(A^)p(B2),  etc.  Obviously,  this  corresponds  to  independent 
choice.  This  leads  to  the  same  results  as  obtained  using  the 
"average"  components;  however,  the  "average"  component  approach' 
requires  only  one  application  of  Monte  Carlo.  The  second  way 
to  obtain  the  probabilities  that  the  various  design  combina¬ 
tions  are  chosen  is  to  obtain  the  joint  probabilities  directly. 

In  terms  of  the  example,  p(A^,  B^),  p(A^,  B2 ) ,  etc.  are  all  ob¬ 
tained  by  direct  questioning. 

(c)  In  practice  there  probably  will  be  some  choices  that  are 
correlated  and  some  that  are  not,  in  which  case  a  mixture  of  meth¬ 
ods  will  be  necessary. 

(d)  The  problem  of  choosing  between  the  various  configura¬ 
tions  can  be  greatly  facilitated  by  knowing  the  system  performance 
probability  distributions  for  each  of  the  configurations.  The 
various  configurations  can  be  compared  on  the  basis  of  expected 
performance,  or  on  the  basis  of  the  probability  of  success  (meet¬ 
ing  or  exceeding  the  specifications).  This  is  an  avenue  of  further 
investigation  that  may  prove  to  be  at  least  as  rewarding  as  the 
present  effort  to  obtain  a  measure  of  progress.  It  really  is  a 
separate  topic,  however,  and  therefore  will  not  be  discussed  fur¬ 
ther  in  this  Memorandum. 

)Hr 

Note  that  this  is  the  definition  of  independence  in  probability 
theory.  See  Ref.  24,  p.  88. 

**Some  difference  between  the  two  may  result  because  of  sampling 
differences . 
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III.  MEASUREMENT  OF  PROGRESS 

This  section  discusses  the  comparison  of  statistical  measures  ob¬ 
tained  at  different  times  as  a  means  of  measuring  progress.  An  example 
is  included  and  some  comments  are  made  about  how  to  present  the  infor¬ 
mation  to  management.  Methods  for  dealing  with  changes  in  contract 
conditions  are  also  discussed. 

TRACKING  CHANGES  IN  UNCERTAINTY 

Measuring  progress  at  the  weapon  system  performance  level  involves: 

(a)  Obtaining  probability  distributions  for  the  critical  system 
performance  characteristics  and  calculating  the  desired  sta¬ 
tistical  measures  as  described  in  the  preceding  section 
(steps  1-4);  and 

(b)  Comparing  the  values  obtained  with  the  values  of  the  same 
measures  obtained  at  an  earlier  date  (step  5). 

The  measures  of  progress  obtained  in  this  way  are  comparative. 

They  indicate  the  relative  amounts  of  progress  made  during  different 

“fc 

time  periods  for  a  given  project.  This  suggests  that  some  pattern  of 
progress  may  be  developed  that  can  be  used  as  a  standard.  However, 
in  this  Memorandum  "tracking"  means  observing  values  of  statistical 
measures  at  different  times.  It  has  nothing  to  do  with  comparing  the 
measures  to  standards  or  budgeted  figures. 

Generally,  progress  is  indicated  by  changes  in  measures  of  central 

tendency,  by  increases  in  probabilities  of  meeting  or  exceeding  specifi¬ 
cations,  and  by  decreases  in  measures  of  dispersion.  At  this  point,  it 
is  important  to  emphasize  that  time  and  cost  must  be  fixed.  The 

* 

It  may  also  be  possible  to  compare  parallel  projects  using  such 
measures.  However,  the  projects  must  be  strictly  parallel  as  regards 
the  time  and  amount  of  money  allowed  and  the  type  of  product  being 
developed. 

SWr 

Development  is  a  process  of  sequential  accumulation  of  knowledge-- 
learning.  Further  research  incorporating  some  concept  of  the  rate  of 
learning  may  be  interesting  (see  Ref.  16,  pp.  13-16). 
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probability  distributions  obtained  at  any  time  are  conditional  upon  the 
time  fnd  money  remaining  to  complete  the  project.  If  the  time  or  the 
money  condition  is  changed,  then  the  probabilities  also  may  change. 

Referring  once  again  to  the  example  of  a  hypothetical  aircraft  de¬ 
velopment  project,  assume  a  minimum  speed  of  675  mph  has  been  stated 
as  a  performance  objective.  This  specification  is  not  to  be  exceeded 

■k 

by  more  than  7.4  percent;  i.e.,  the  maximum  acceptable  speed  is  725  mph. 

For  purposes  of  illustration,  assume  the  time  period  for  the  contract 

is  two  years  and  the  price  is  $100  million.  Table  4  shows  probability 

distributions,  p(V),  derived  at  3-month  intervals,  for  the  speed  of  the 
** 

aircraft.  The  data  represent  the  history  of  the  development  up  to 
the  final  three  months;  thus,  no  data  are  shown  for  the  end  of  the  proj¬ 
ect.  At  the  end  of  the  final  period,  the  aircraft  will  be  test  flown, 
and  its  speed,  under  anticipated  operational  conditions,  will  be  known 
with  certainty.  Table  5  illustrates  the  summary  statistics  and  time 
and  cost  information  required  to  evaluate  progress  and  status.  The  sta¬ 
tistical  me  sures  in  Table  5  were  derived  from  the  data  in  Table  4  (using 
the  definitions  in  Appendix  D) . 

The  probabilities  in  To’-’j  4  chosen  co  illustrate  the  importance 

of  using  more  than  one  of  the  statistical  measures  as  an  indicator  of 
P'-ogr^ss  or  status.  This  is  apparent  at  three  different  times.  First, 
in  going  from  time  t  =  9  mo  to  time  t  =  12  mo,  ct(V)  shows  a  slight  in- 
crease.  Taken  by  itself,  this  would  indicate  "negative"  progress. 


k 

Savings  in  cost  or  time  are  preferable  to  attaining  a  speed  greater 
than  725  mph. 

■kk 

These  distributions  are  the  output  from  the  Monte  Carlo  process. 
The  figures  need  not  be  for  equal  time  intervals  as  shown. 

•kick" 

Negative"  progress  may  be  an  inappropriate  label  for  such  sit¬ 
uations.  Increases  in  the  standard  deviation,  or  an  alternative  mea¬ 
sure  of  dispersion,  can  occur  for  a  number  of  different  reasons.  In 
the  example,  there  was  an  upward  shift  of  the  entire  probability  dis¬ 
tribution.  This  showed  up  as  an  increase  in  the  probabilities  of  higher 
performance  capability.  It  also  resulted  in  the  distribution  being 
more  spread  out;  hence  a  larger  dispersion.  An  increase  in  dispersion 
may  also  result  from  the  discovery  of  an  unforeseen  problem;  however 
discovering  this  amounts  to  "positive"  progress  in  a  learning  sense 
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Tublo  4 

PROBABILITY  DISTRIBUTIONS  FOR  SPEED  OF  HYPOTHETICAL  AIRCRAFT 
AT  THREE-MONTH  INTERVALS  DURING  DFVEIGPMENT 


g\t^ 

0  mo 

3  mo 

6  mo 

9  mo 

12  mo 

15  mo 

18  mo 

21  mo 

650  mph 

.25 

.20 

.  15 

.10 

.05 

.05 

.05 

o 

o 

• 

675  mph 

.30 

.30 

.30 

.30 

.25 

.20 

.20 

700  mph 

.25 

.30 

.35 

.35 

.40 

.50 

.55 

725  mph 

.25 

.25 

.25 

.25 

.25 

.20 

.20 

750  mph 

.00 

.00 

.00 

.00 

.05 

.05 

.05 

.05 

Table  5 


INFORMATION  FOR  EVALUATION  OF  PROGRESS  AND  STATUS  OF  HYPOTHETICAL 
AIRCRAFT  DEVELOPMENT  PROJECT  SHOWN  IN  TABLE  4 


^Cein 

0  mo 

3  mo 

6  mo 

9  mo 

12  mo 

15  mo 

18  mo 

21  mo 

P(V)  •  675) 

.75 

.80 

.85 

.90 

.95 

.95 

.95 

1.00 

P(V  >  725) 

.00 

.00 

o 

o 

« 

.05 

.05 

.05 

.05 

u(V)  (mph) 

688 

689 

691 

694 

699 

700 

700 

703 

A(V)(mph) 

2‘» .  7 

28.6 

26.1 

25.6 

26.3 

25.9 

22.4 

19.2 

Dollars 

consumed 

Negli- 

6 

10 

12 

13 

13 

15 

15 

(millions) 

gible 

Time  in¬ 
to  rva  1  (mo) 

0 

3 

3 

3 

3 

3 

3 

3 

Dollars  re¬ 
maining 
(mi llions) 

100 

94 

84 

72 

59 

46 

31 

16 

Time  remain¬ 
ing  (mo) 

24 

21 

18 

15 

12 

9 

6 

3 

Also,  the  probability  of  being  in  the  acceptable  range  shows  no 
progress.  However,  both  p(V  "  675)  and  p.(V)  increase,  indicating  prog¬ 
ress.  Second,  from  time  t  -  12  mo  to  time  t  -  15  mo,  p(V  >  675)  shows 
no  change,  but  both  p-(V)  and  a(V)  show  slight  improvements.  Third,  from 
time  t  -  15  mo  to  time  t  -  18  mo,  p(V  "*  675)  and  u(V)  do  not  change,  indi¬ 
cating  no  progress.  However,  a(V)  does  indicate  progress. 
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It  has  been  assumed  in  the  example  that  the  first  time  subjective 
probabilities  are  collected  and  the  probability  distributions  for  the 
system  performance  characteristic  are  generated,  the  amount  of  time  and 
money  consumed  are  negligible  in  comparison  to  the  requirements  for  the 
total  project.  This  will  not  always  be  the  case. 

The  example  not  only  illustrates  the  need  to  look  at  more  than  one 
measure  of  progress;  it  also  indicates  that  the  cause  of  difficulty 
cannot  be  determined  from  the  summary  statistical  measures.  Thus,  the 
method  can  be  thought  of  as  a  tool  for  management  by  exception.  An  in¬ 
dication  of  unsatisfactory  progress  is  a  signal  to  look  deeper  into  the 
project  to  determine  the  cause.  The  probability  distributions  for  the 
characteristics  of  the  component  parts  are  the  first  place  to  look. 

Examination  of  these  distributions  will  indicate  which  component,  or 

* 

components,  to  analyze  further. 

Graphical  Presentation 

The  statistical  measures  may  be  displayed  graphically  in  several 
ways.  Figure  5  shows  the  probability  of  meeting  or  exceeding  the  min¬ 
imum  speed  requirement  (the  design  specification),  and  the  probability 
of  being  within  the  maximum  and  minimum  specifications.  Figure  6  shows 
the  magnitude  of  the  standard  deviation  of  the  distributions  for  speed. 
Figure  7  shows  the  mean,  p,  of  the  distributions  and  the  mean  plus  and 
minus  the  standard  deviation,  p  +  o. 

In  Fig.  6,  for  example,  if  the  magnitude  of  the  standard  deviation 
in  mph  is  taken  as  a  measure  of  uncertainty,  the  amount  of  uncertainty 
remaining  (status)  can  be  measured  at  any  given  time.  The  distance 
from  the  horizontal  dashed  line  down  to  the  standard  deviation  line  is 
the  total  amount  of  uncertainty  removed.  Thus,  at  time  t  =  9  mo,  the 
amount  of  uncertainty  removed  is  4.1  mph,  and  the  time  remaining  is 


The  search  for  underlying  causes  of  difficulty  would  be  facili¬ 
tated  by  a  list  of  factors  that  the  individual  engineers  considered  to 
be  important  when  they  were  questioned  to  determine  their  probabilities. 
This  suggests  that  a  checklist  of  factors  frequently  considered  may  be 
an  appropriate  adjunct  to  any  input  sheet  developed  for  recording  en¬ 
gineers'  subjective  probabilities. 


Magnitude  of  the  st-a.dard  deviation  (mp'n> 


0  3  6  9  12  15  18  21  2‘ 

Time  (mo) 

Fig.  5  -  Probability  of  Meeting  oi  Exceeding  Minimum  Requirement  and 
Piobability  of  Being  within  Maximum  and  Minimum  Requirement  Limits 


Fi.i.  6  -  Magnitude  of  Standard  Deviation  at  Three- Month  intervals 
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Speed  (mph) 
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Time  (mol 

Fiq.  7  -  Mean  Plus  and  Minus  Standard  Deviation  at  1  inee-Month  Intervals 


Lb  mo.  A  diagonaL  dotted  Line  is  shown  trom  time  21  mo  to  the  time 
axis  at  time  t  =  24  mo  to  indicate  that  after  the  aircraft  is  tested, 
under  the  anticipated  operational  conditions  there  will  be  no  uncer¬ 
tainty  regarding  its  rated  speed. 


THE  EFFECT  OF  CHANGES  IN  THE  CONTRACT 

In  the  discussion  above,  it  has  been  assumed  that  the  time,  cost, 
and  technical  objectives  of  a  development  project  remain  fixed.  This 
is  rarely  true  in  reality.  Changes  in  any  of  the  terms  of  a  contract-- 

Vc 

time,  cost,  or  performance  --will  result  in  different  appraisals  of  how 


Changes  in  specifications  as  a  source  of  uncertainty  have  been 
examined  empirically  by  II.  B.  Eyring  (see  Ref.  10).  His  findings  sug¬ 
gest  that  contractor  personnel  view  requirements  (specifications)  as 
being  flexible.  Hence,  there  is  not  only  uncertainty  about  what  can 
be  done  but  also  about  what  the  customer  wants. 
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the  project  will  turn  out.  This  is  so  because  the  subjective  probabil¬ 
ities  obtained  at  any  given  time  reflect  not  only  the  knowledge  at  that 
time  but  also  the  time  and  money  remaining  to  complete  the  job  and  the 
management  and  organization  of  the  project.  In  other  words,  the  subjec¬ 
tive  probabilities  depend  on  the  conditions  at  the  time  the  probabilities 

are  obtained.  If  changes  are  made  in  any  of  the  conditions,  then  the 

* 

probabilities  may  also  change. 

Changes  in  the  time,  cost,  or  specifications  may  be  precipitated 

by  changes  in  some  source  of  uncertainty  that  is  external  to  the  con- 

** 

tractor--for  example,  uncertainty  regarding  enemy  capabilities.  Re¬ 
gardless  of  the  source  of  such  changes,  when  it  is  important  that  their 
impact  on  the  measurement  of  status  and  progress  be  distinguished  from 
the  gains  in  technical  knowledge  made  by  the  contractor,  then  the 
approach  described  below  can  be  used. 


Time-Cost  Changes 

It  is  assumed  that  changes  in  allocation  are  determined  at  mana¬ 
gerial  levels  above  the  person  (the  working- level  engineer)  who  pro¬ 
vides  probability  estimates.  Thus,  the  engineer  is  not  a  direct  partic¬ 
ipant  in  achieving  any  changes  in  the  time  or  money  available.  However, 
a  change  in  resources  will  usually  have  an  impact  on  potential  perform¬ 
ance  and  will  require  new  probability  estimates  from  the  engineer. 

The  two  sources  of  change  in  uncertainty  can  be  isolated  by  ask¬ 
ing  the  engineer  for  two  sets  of  subjective  probabilities/  One  set 


Both  component  characteristic  probabilities  and  design  choice 
probabilities  are  subject  to  change. 

** 

Ref.  14,  pp.  15-16. 

"kick 

This  distinction  suggests  the  use  of  the  technique  described 
in  this  Memorandum  as  a  means  for  comparing  alternative  resource  allo¬ 
cation  schemes.  The  government  agency  can  select  one  of  the  alterna¬ 
tives  on  the  basis  of  time,  cost,  and  expected  performance.  This  also 
may  be  done  during  the  project  to  determine  the  effect  of  proposed 
changes  in  resource  allocation  on  expected  performance. 

+ 

There  are  arguments  for  and  against  informing  the  engineer  of  the 
changes  in  allocation  before  he  is  questioned  to  determine  his  proba¬ 
bilities  under  the  old  allocation.  The  issue  is  not  clear.  Furthermore, 
as  a  practical  matter,  there  usually  is  substantial  advance  notice  of 
proposed  changes  . 
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would  ba  obtained  for  the  old  allocation  pattern  and  the  other  set  for 
the  now.  The  probabilities  associated  with  the  old  allocation  yield 
uncertainty  measures  that  reflect  the  technical  knowledge  gained  to 
that  time.  The  probabilities  associated  with  the  new  allocation  re¬ 
flect  both  the  technical  knowledge  and  the  change  in  allocation.  Thus 
the  difference  between  measures  for  the  two  sets  of  probabilities  in¬ 
dicates  the  change  In  uncertainty  associated  with  the  change  in  allocation. 

Continuing  with  the  aircraft  development  example,  suppose  that  at 
the  end  of  '21  months  the  contractor  is  informed  that  he  has  been  given 
an  additional  $30  million  and  6  months  to  complete  the  project.  The 
contractor  notifies  the  engineers  involved  and  a  new  set  of  subjective 
probability  distributions  is  obtained.  Suppose  that  the  probability 
distribution  for  the  speed  of  the  aircraft  under  the  now  conditions  is 
p(V  “  675  mph)  *  .10,  p(V  "  700  mph)  31  .60,  p(V  *  725  mph)  «  .25,  and 

p(V  *  750  mph)  *  .05.  The  associated  statistical  measures  are  then 

p(V  >  675  mph)  *  1.00,  p(V  >  725  mph)  «  .05,  u(V)  1  706  mph,  and  a(V)  = 

+  17.4  mph.  Figure  8  shows  the  "history”of  the  standard  deviation  for 
the  project  as  in  Fig.  6  In  thin  lines.  The  bold-face  addition  depicts 


Fig.  8  -  Standard  Deviation  Assuming  a  Change  in  Time  and  Money  Available 
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tlie  new  conditions.  The  change  in  the  magnitude  of  the  standard  devi¬ 
ation  from  22.4  mph  to  19.2  mph  is  due  to  the  technical  knowledge  gainea 
by  the  contractor.  The  additional  change  from  19.2  mph  to  17.4  mph  is 
due  to  tlie  change  (increase)  in  time  and  money  available. 

Specifications  (Requirements)  Changes 

Changes  in  the  performance  requirements  of  a  weapon  system  will 
also  have  an  Influence  on  the  amount  of  uncertainty.  The  situation 
is  similar  to  a  change  in  the  time-cost  conditions.  Two  sets  of  prob¬ 
ability  distributions  must  he  obtained.  The  difference  between  the  ef¬ 
fect  of  the  guin  in  technical  knowledge  and  the  effect  of  the  change 
in  specifications  can  be  determined  as  in  the  discussion  of  time-cost 
changes  above . 

Combined  Changes 

When  both  a  specifications  change  and  a  time-cost  change  are  made, 
the  effect  of  the  combined  change  can  be  separated  from  the  effect  of 
gains  in  technical  knowledge,  again  by  the  same  means  as  described  above. 

There  may  be  situations  in  which  it  is  desired  to  separate  the 
effect  of  the  specifications  change  from  the  effect  of  the  time-cost 
change.  This  can  be  done  by  obtaining  another  set  of  probability  dis¬ 
tributions  for  one  of  the  changes;  e.g.,  for  the  change  in  specifica¬ 
tions,  or  for  the  change  in  time-and-cost .  Suppose  that  the  specifi¬ 
cations  change  is  used  as  the  basis  for  the  third  set.  Then,  the  change 
due  to  the  time-cost  change  is  given  by  the  difference  between  the  new 
probability  distributions  based  on  both  changes  and  the  new  probabil¬ 
ity  distributions  based  on  only  the  specifications  change;  the  change 
due  to  the  specifications  change  is  given  by  the  difference  between 
the  new  probability  distributions  based  on  only  the  specifications 
change  and  the  new  probability  distributions  assuming  no  contract 
changes;  and,  the  change  due  to  the  gain  in  technical  knowledge  is 
given  by  the  difference  between  the  new  probability  distributions  as¬ 
suming  no  contract  changes  and  the  old  (previously  obtained)  proba¬ 
bility  distributions.  Of  course,  obtaining  another  set  of  subjective 
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probabilities  and  carrying  out  the  Monte  Carlo  another  time  requires 
more  effort.  Furthermore,  there  is  no  apparent  reason  for  choosing  be¬ 
tween  the  specifications  change  and  the  time-cost  change  for  the  basis 
of  the  third  set  of  probabilities,  and  there  is  no  reason  to  expect 
that  the  results  will  be  independent  of  the  choice  made. 

The  problems  of  obtaining  an  additional  set  of  probability  dis¬ 
tributions  and  of  deciding  which  of  the  change  categories  to  choose  as 
a  basis  for  obtaining  the  third  set  of  probabilities  is  eliminated  if 
the  change  in  specifications  does  not  induce  the  contractor  to  make  any 
changes.  Under  this  assumption,  the  only  measures  that  are  affected 
are  those  that  relate  directly  to  the  specifications.  Note  that  mea¬ 
sures  of  central  tendency  and  dispersion  are  related  to  the  specifica¬ 
tions  only  through  the  contractor's  choice  of  design  or  management  ap¬ 
proach;  hence,  if  the  approach  is  not  modified,  then  the  measures  of 
central  tendency  and  dispersion  will  not  be  changed.  The  uncertainty 
measures  that  are  affected  are  the  probability  of  meeting  or  exceeding 
the  specification  and  the  probability  of  being  within  the  maximum  and 
minimum  specification  limits. 

Again  referring  to  the  aircraft  example,  suppose  that  at  21  mo 
there  are  time  and  cost  changes  and  also  an  increase  in  the  minimum 
specification  from  675  mph  to  700  mph.  The  probability  of  meeting  or 
exceeding  the  specification  becomes  p(V  >  700  mph)  =  .90.  To  determine 
the  effect  of  the  specification  change  only,  use  the  probability  dis¬ 
tribution  for  the  old  allocation  and  subtract  the  probability  of  meeting 
or  exceeding  the  new  specification.  For  the  example  the  change  is 
.80  -  1,00  =  -.20.  To  determine  the  effect  of  the  allocation  change 
only,  next  subtract  the  probability  of  meeting  or  exceeding  the  new 
specification  using  the  probability  distribution  for  the  new  allocation 
from  the  probability  of  meeting  or  exceeding  the  new  specification  using 


The  results  of  this  assumption  are  identical  to  the  case  of  de¬ 
termining  the  effect  of  a  contemplated  change  in  specifications  using 
the  distributions  based  on  -  1  ange  and  the  distributions  based  on  the 

time-cost  change. 


Prcbobihty 
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Fig.  9  -  Probability  of  Meeting  or  Exceeding  Specification 
Assuming  Changes  in  Specification  and  Time  and  Money 


the  probability  distribution  for  the  old  allocation.  For  the  example, 

the  result  is  .90  -  .80  =  +.10.  The  net  effect  is  thus  a  decrease  in 

the  probability  of  success  of  -.10  (from  1.00  to  .90).*  This  is  shown 
*■* 

in  Fig.  9. 

In  deciding  whether  to  distinguish  between  gains  in  technical 
knowledge  and  changes  in  the  contract  the  following  considerations 
should  be  kept  in  mind: 


In  this  example  the  two  types  of  change  are  partially  compensating. 
They  can  also  be  additive. 

irk 

The  reader  should  note  that  different  results  for  the  individual 
changes  arc  obtained  if  the  order  o*  determination  is  switched.  This 
is  because,  in  this  case,  the  probability  of  meeting  or  exceeding  the 
old  specification  is  1.00  for  the  old  allocation.  An  increase  in  time 
and  money  will  not  raise  this  figure.  In  those  cases  where  the  prob¬ 
ability  of  success  i s  not  already  1.00,  the  results  wi’l  be  the  same 
for  each  "partial"  effect  regardless  of  the  order. 
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(1)  In  order  to  isolate,  the  effect  of  each  type  of  change  it  is  neces¬ 
sary  to  gather  two  sets  of  subjective  probabilities  and  carry  through 

* 

two  Monte  Carlo  simulations.  Because  this  involves  additional  effort, 
it  is  Important  that  the  recipient  of  the  final  progress  data  have  a 
need  for  such  a  breakout.  If  he  is  not  interested  in  the  separate  ef¬ 
fects  on  the  progress,  then  it  is  only  necessary  to  obtain  figures  for 
the  new  conditions.  (2)  Additionally,  during  a  development  project 
there  are  myriad  detailed  engineering  changes  that  taken  individually 
have  very  small  significance  to  system  performance.  If  these  changes 
were  subjected  to  the  analysis  described  above,  almost  continuous  data 
collection  and  analysis  would  be  required.  Hence,  it  appears  prudent 
to  distinguish  between  change  in  technical  knowledge  and  contract 
changes  only  when  the  contract  change  is  a  major  one. 


This  is  for  the  case  when  specifications  changes  by  themselves 
do  not  induce  the  contractor  to  alter  his  design  or  management  approach. 
When  specifications  changes  do  induce  alterations  and  it  i3  desired  to 
distinguish  between  the  effects  of  specifications  changes  and  time-and- 
money  changes  in  addition  to  the  gains  in  technical  knowledge,  then  the 
analysis  and  calculations  become  more  complicated.  Furthermore,  the 
arbitrary  decision  about  which  individual  contract  change  to  consider 
first  influences  the  result.  Consequently,  under  normal  circumstances 
such  analyses  probably  should  not  be  attempted. 
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IV.  CONCLUSIONS  AND  SUGGESTIONS  FOR  OTHER  APPLICATIONS 

The  motivation  for  the  work  described  in  this  Memorandum  has  been 
to  devise  a  means  ot  obtaining  better  information  on  progress  toward 
meeting  technical  performance  objectives  in  weapon  system  development 
projects.  The  approach  makes  use  of  some  of  the  important  character¬ 
istics  of  weapon  system  development  and  decision  theory.  These  char¬ 
acteristics  are  described  in  tne  Introduction. 

The  procedure  that  evolved  uses  a  five-step  framework.  Eriefly, 
the  five  steps  are:  (1)  obtain  design  equations;  (2)  determine  subjec¬ 
tive  probabilities;  (3)  generate  probability  distributions  for  system 
performance;  (4)  calculate  statistical  measures;  and  (5)  compare  meas¬ 
ures  at  different  times. 

The  first  four  steps  lead  to  quantitative  measures  of  uncertainty 
and  figures  for  the  probability  of  meeting  or  exceeding  contract  re¬ 
quirements  (specifications).  The  latter  quantity  is  a  numerical  state¬ 
ment  of  the  chances  of  success.  The  fifth  step  gives  a  number  which 
indicates  progress,  or  the  lack  of  it.  The  resulting  numbers  are  in¬ 
dicators  for  management  by  exception.  If  the  figures  are  unsatisfac¬ 
tory,  then  the  probability  distributions  for  component  characteristics 
should  be  examined  to  discover  the  source,  or  sources,  of  difficulty. 

This  search  would  be  greatly  facilitated  by  data  regarding  the  factors 
considered  by  the  engineers  at  the  time  of  determination  of  probabilities. 

The  amount  of  uncertainty  is  influenced  by  the  conditions  of  the 
contract- -  time ,  cost,  and  performance  requirements.  Any  changes  in 
these  conditions  will  therefore  change  the  uncertainty.  A  means  of 
accounting  for  such  changes  has  been  outlined. 

Implementation  of  such  a  technique  should  be  preceded  by  testing 
of  methods  for  determining  subjective  probabilities.  The  first 
application  should  be  considered  experimental.  In  a  very  real  sense, 
the  only  "laboratory"  that  is  suitable  for  testing  the  procedure  is  a 
system  development  project.  The  procedure  has  little  utility  unless 
it  actually  can  be  applied  to  such  long-term,  non-random  processes 
characterized  by  the  sequential  accumulation  of  information. 
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Three  other  uses  of  the  general  approach  are  suggested  for  future 
development:  (1)  Comparison  of  the  status  and  progress  of  parallel 

projects  would  provide  information  helpful  in  selection  among  contrac¬ 
tors.  (2)  Comparison  of  various  resource  allocation  schemes  for  one 
project  would  provide  information  on  the  chances  of  success  for  several 
patterns  of  resource  allocation--information  that  the  contractor  or 
the  government  could  use  to  evaluate  and  choose  between  the  possible 
allocation  schemes.  (3)  Generation  of  a  set  of  distributions  for  the 
overall  system  for  each  of  several  alternate  components  would  pro¬ 
vide  probabilistic  information  useful  to  the  contractor  or  government 
in  selecting  between  competing  components. 
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Appendix  A 

SYSTEM  DEVELOPMENT  VIEWED  AS  A  DECISION  AND  INFORMATION  PROCESS 

In  weapon  system  development  there  is  no  unique  function  that  de¬ 
termines  the  utility,  or  value,  of  the  end  item;  hence,  decisionmakers 
cannot  choose  courses  of  action  that  will  lead  to  the  "best"  configura¬ 
tion  for  the  end  item.  Instead,  the  decisionmakers  must  choose  actions 
that  will  lead  to  "satisfactory"  results.  Decisionmaking  under  these 
conditions  has  been  referred  to  by  theoreticians  as  "satisficing," 

ie 

and  the  desired  results  are  called  "levels  of  aspiration."  The  process 
of  decisionmaking  involves  a  comparison  of  results  with  aspirations, 
and  a  selection  of  a  course  of  action.  The  choice  depends  on  whether 
or  not  the  comparison  is  satisfactory. 

In  weapon  system  development,  there  is  periodic  review.  The  re¬ 
sults  achieved  between  review  points  are  measured  by  progress  toward 
the  technical  objectives.  The  results  achieved  at  the  end  of  the  proj¬ 
ect  are  values  for  the  variables  in  the  technical  objectives.  Hence, 
during  the  project,  the  aspiration  levels  must  relate  to  how  much  prog¬ 
ress  is  satisfactory,  while  at  the  end  of  the  project,  the  aspiration 
levels  are  concerned  exclusively  with  the  final  values  for  the  perform¬ 
ance  characteristics. 

Figure  10  shows  a  simple  satisficing  model  for  the  top  management 
portion  of  the  decision  and  information  process  for  weapon  system  de¬ 
velopment.  The  information  and  decision  interface  between  the  military 
and  a  defense  contractor  is  indicated  in  the  figure  by  a  dotted  line. 

The  model  is  presented  here  to  show  the  relationship  of  the  procedure 
described  in  the  text  to  the  military  and  the  contractor.  The  proce¬ 
dure  in  the  text  is  concerned  only  with  boxes  10  through  13  (outlined 
with  a  heavy  dashed  line). 


For  example,  see  Ref.  30  and  Ref.  11,  pp.  632-634.  A  simple  model 
is  shown  in  Ref.  21,  pp.  48-49. 

A  more  accurate  distinction  is  between  the  review  points  when 
the  entire  system  can  be  tested  to  determine  values  for  all  the  variables 
in  the  technical  objectives,  and  the  points  when  it  cannot. 
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Fig.  10  -  A  View  of  the  Decision  and  Information  Process  for  Weapons  System  Development  Projects 
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The  model  offers  many  opportunities  for  further  development.  For 
example,  the  process  of  setting  aspiration  levels  (boxes  2  and  3)  re¬ 
quires  additional  research.  Although  current  decisionmaking  indicates 
that  aspiration  levels  exist  and  are  used,  the  process  by  which  they 
are  set  has  not  been  thoroughly  investigated. 


See  Ref.  6,  Chap.  3. 
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Appendix  B 

DETERMINATION  OF  SUBJECTIVE  PROBABILITIES 


This  Appendix  describes  the  nature  of  subjective  probabilities  and 
a  few  simple  methods  that  may  be  used  to  determine  an  individual’s  sub¬ 
jective  probabilities.  In  addition,  some  of  the  factors  that  may  in- 
fluence  the  "quality"  of  the  numbers  obtained  are  discussed. 


•ft  f  ' 

The  personal  or  subjective  view  of  probability  is  relatively  new. 

In  spite  of  the  eminent  reputation  and  growing  number  of  people  who  ac¬ 
cept  the  subjective  view  of  probability  as  valid,  there  are  many  intel¬ 
ligent  and  reputable  scholars  and  practitioners  who  do  not  believe  that 
subjective  probabilities  are  legitimate  in  and  of  themselves;  and  there¬ 
fore  not  legitimate  as  aids  to  decisionmaking.  In  fact,  there  is  a 
heated  controversy  over  the  nature  of  probabilities  and  there  are  sev¬ 
eral  schools  of  thought.  (See,  for  example.  Ref.  28.)  The  major  issue 
is  concerned  with  the  nature  of  probabilities.  The  probabilities  in 
this  Memorandum  are  personal  (individual)  beliefs  regarding  the  ultimate 
occurrence  of  an  event  which  is  in  general  not  the  result  of  a  random 
process.  The  probabilistic  school  of  thought  which  is  most  opposed  to 
this  personal  view  of  probabilities  is  the  objective  school.  The  ob- 
jectivists  believe  that  probabilities  have  meaning  only  when  they  are 
related  to  random  events,  such  as  drawing  a  card  from  a  shuffled  deck, 
throwing  a  die,  etc.  A  recent  expression  of  the  view  that  the  use  of 
subjective  probabilities  is  not  legitimate  was  given  by  Kenneth  E.  Boulding 
(Ref.  3,  p.  B-163).  It  is  true  that  assignment  of  subjective  probabil¬ 
ities  to  non-random  events  is  a  substitute  for  real  understanding  of 
"  ...  the  truly  systematic  nature  of  the  world  ...."  (Ref.  3,  p.  B-163.) 

It  is  also  true  that  better  decisionmaking  would  result  if  the  real  un¬ 
derstanding  existed.  However,  Boulding  seems  to  miss  the  important 
point  that  many  real  world  phenomena  are  so  complex  that  even  if  a  model 
could  be  built  which  reflected  every  detail,  it  would  still  be  impos¬ 
sible  to  use  the  model  because  of  the  time  involved  in  assembling  the 
required  input  information  and  performing  whatever  analyses  are  required 
to  get  the  "answer."  In  these  cases,  reliance  on  an  exact  model  will 
be  impossible.  Subjective  probabilities  are  used  to  capture  the  "best" 
information  possible  given  the  time  constraint--whether  there  is  an 
exact  model  or  not. 

Several  good  references  on  subjective  probabilities  are:  Ref.  28, 
Chaps.  1,  3,  and  4;  Ref.  29,  Chaps.  1,  3,  and  12;  Ref.  12,  Chap.  9,  es¬ 
pecially  pp.  253-263;  Ref.  19,  Chap.  13;  Ref.  2,  Chap.  1;  Ref.  7;  Ref. 

18;  and  Ref.  20.  The  last  reference  is  the  most  recent  survey  of  sub¬ 
jective  probabilities  and  decisionmaking.  It  contains  an  extensive 
bibliography,  including  publications  by  nearly  all  prominent  researchers 
in  the  field. 
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POSSIBILITIES  AND  PROBABILITIES; 

THE  NATURE  OF  SUBJECTIVE  PROBABILITIES 

Subjective  probabilities  are  numbers  that  are  subjective  in  the 

if 

sense  that  they  express  the  beliefs  of  an  individual  person  and  prob¬ 
abilities  in  the  sense  that  they  satisfy  the  axioms  of  probability 
** 

theory. 

There  are  many  activities  undertaken  by  man  that  have  several  pos¬ 
sible  outcomes  (possibilities).  When  an  individual  must  make  decisions 
under  such  conditions  he  usually  enumerates  the  possibilities  as  part 
of  his  analysis.  In  addition,  the  decisionmaker  must  have  an  assess¬ 
ment  of  the  chances  (probabilities)  that  each  of  the  possibilities  will 
occur.  There  are  several  common  schemes  for  the  assessment  of  chance. 

Some  examples  are  "  ...  evaluative  phrases  (good,  fair,  not  so  hot,  etc.), 
numerical  rankings  (I,  II,  III),  alphabetical  rankings  (A,  B,  C,  D),  or 
numerical  odds  (1  in  5)."  The  most  extensive  mathematical  analyses 
can  be  made  if  the  assessment  is  expressed  in  terms  of  probabilities. 

The  statement  of  any  subjective  judgment  obviously  presupposes  some 
knowledge  of  the  subject  by  the  individual.  The  source  of  knowledge  is 
past  experience  with  similar  or  analogous  events.  The  knowledge  may  be 
personal  or  it  may  be  obtained  from  the  experience  of  others.  It  may 
be  in  the  form  of  extensive  statistical  compilations,  or  it  may  be  frag¬ 
mentary  bits  of  information.  There  is  nearly  always  some  information 
available. 

Given  that  some  information  is  available  on  which  individuals  can 
base  their  subjective  judgments  and  that  it  is  desirable  to  have  these 
judgments  stated  in  the  form  of  probabilities,  it  is  necessary  to  ex¬ 
amine  various  ways  to  get  persons  to  state  their  judgments  in  probabil¬ 
istic  terms. 

*Ref .  27,  p.  3. 

if  ic 

Ref.  24,  p.  18. 

***Ref.  12,  p.  255. 
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TBCHNIQUES 

Techniques  for  determining  subjective  probabilities  can  be  identi¬ 
fied  with  one  of  the  following  three  categories,  as  distinguished  by 

„  * 

L.  J.  Savage: 

1.  Question  the  person  directly;  ask  him  what  probability  he  at¬ 
taches  to  the  outcome  of  a  particular  event.  Although  this  is  the  sim¬ 
plest  way,  it  is  the  least  likely  to  work  because  many  individuals  do 
not  think  directly  in  terms  of  probabilities.  Furthermore,  those  who 
do  think  in  probabilities  may  have  difficulty  communicating  them  with¬ 
out  the  aid  of  some  auxiliary  device. 

2.  Observe  the  actions  of  the  person  whose  probabilities  it  is 
desired  to  ascertain.  This  technique  cannot  be  used  in  the  application 
considered  in  this  Memorandum  for  two  reasons:  First,  it  gives  infor¬ 
mation  after  the  fact,  and  prior  information  is  required.  Second,  it 
does  not  assign  numbers  to  the  probabilities.  It  only  indicates  the 
event  with  the  highest  probability. 

3.  Ask  the  person  to  make  a  choice  between  different  "betting" 
situations.  This  may  be  done  by  using  techniques  that  will  give  the 
probability  density  function  or  by  using  techniques  that  will  give  the 
probability  distribution  function  (see  Appendix  D).  Which  probability 
function  is  better  depends  on  the  use  to  which  it  will  be  put.  Drawing 
random  samples  is  most  easily  accomplished  using  the  distribution  func¬ 
tion.  Calculation  of  statistical  measures  is  done  using  the  density 
function.  The  merits  of  the  techniques  described  here  are  related 

to  the  subject's  ability  to  respond  easily  and  intelligently,  and  to 
the  confidence  that  can  be  placed  in  his  responses.*** 


Ref.  27,  pp.  26-30. 

This  presents  no  problem,  however,  because  the  density  function 
can  be  obtained  from  the  distribution  function  by  subtraction,  and  the 
distribution  function  from  the  density  function  by  addition. 

Confidence  is  higher  for  this  third  class  of  techniques  because 
of  the  greater  "ease"  of  responding  and  also  because  the  results  derive 
from  the  individual's  actions  of  choosing,  not  from  his  statements  of 
numbers  that  he  believes  to  be  probabilities. 


The  responses  obtained  must  satisfy  the  axioms  of  probability 

theory,  regardless  of  the  technique  used.  If  the  events  are  mutually 

* 

exclusive  and  collectively  exhaustive,  the  conditions  on  the  responses 
can  be  summarized  in  the  following  rules: 

Rule  1.  The  sum  of  the  weights  assigned  to  any  set  of  mutually 
exclusive  and  collectively  exhaustive  events  shall  be  1. 

Rule  2.  The  weight  assigned  to  any  event  shall  be  a  number  be" 
tween  0  and  1,  inclusive,  0  representing  complete  conviction  that 
the  event  will  not  occur  and  1  representing  complete  conviction 
that  it  will  occur. 

Rule  3.  If  two  or  more  mutually  exclusive  events  are  grouped  into 
a  single  event,  the  weight  attached  to  this  single  event  shall  be 
equal  to  the  sum  of  the  weights  attached  to  the  original  events.** 


Determination  of  the  Density  Function 


Choice  between  Gambles 


In  all  "choice  between  gambles"  modes  of  probability  determination, 
the  person  whose  probabilities  are  being  obtained  (hereafter  referred 
to  as  the  engineer)  is  confronted  with  e  choice-making  situation.  He 
is  requested  to  choose  between  a  wager  on  the  outcome  of  the  project 
in  question  (which  will  be  called  the  real  world  gamble)  and  a  wager 
on  a  hypothetical  gamble.  The  hypothetical  gamble  involves  an  objec¬ 
tive  event  with  given  objective  probabilities.  The  real  world  gamble 
involves  the  outcome  or  outcomes  of  the  project  with  unspecified  prob¬ 
abilities.  The  unspecified  probabilities  are  to  be  determined  by  in¬ 
ference.  The  probabilities  of  the  hypothetical  gamble  are  varied  until 


See  Ref.  29,  pp.  7-9,  for  a  discussion  of  mutually  exclusive 
events.  If  the  events  are  not  so  defined,  then  multivariate  techniques 
must  be  used. 

'if'/c 

Ref.  29,  pp.  10-13.  These  rules  are  the  same  as  the  axioms  of 
probability  theory  given  in  Ref.  24,  p.  18.  It  should  be  noted  that 
these  rules  are  valid  for  finite  sample  description  spaces.  When  the 
sample  description  spaces  are  not  finite,  certain  minor  changes  must 
be  made.  See  Ref.  24,  p.  21.  These  changes  do  not  significantly 
affect  the  mettiod  described  in  the  text  of  this  Memorandum. 

Discussion  of  techniques  other  than  those  described  here  can  be 
found  in  Refs.  26  and  31.  Toda  [ 3 l ]  has  developed  some  very  sophisti¬ 
cated  techniques  that  may  be  quite  useful. 
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the  engineer  is  indifferent  between  the  two  gambles.  At  this  point  it 
is  inferred  that  the  engineer's  subjective  probabilities  regarding  the 
outcomes  of  the  real  world  gamble  are  the  same  as  the  stated  probabil¬ 
ities  for  the  hypothetical  gamble. 

Consider  the  case  of  determining  the  probabilities  of  various  pos¬ 
sible  values  for  the  thrust  of  a  jet  engine  that  is  being  developed. 

The  questioning  involves  presenting  the  engineer  with  a  series  of  choices 
of  the  form  shown  in  Table  6.  To  begin,  the  engineer  is  asked  to  state 
a  possible  value  for  the  thrust  of  the  completed  engine.  Suppose  the 
value  he  gives  is  36,000  lb.  This  value  is  entered  in  Table  6  as  shown. 


Table  6 

A  CHOICE-MAKING  SITUATION  TO  DETERMINE  THE  PROBABILITY 
DISTRIBUTION  FOR  THE  THRUST  OF  A  JET  ENGINE  UNDER  DEVELOPMENT 


Real  World  Gamble 

Hypothetical  Gamble 

Consequence 

$10 

$0 

Consequence 

$10 

$0 

Thrust 

36,000  lb 
+  1000  lb 

Anything 

else 

Event 

X 

Y 

Probabilities 

7 

7 

Probabilities 

.7 

.3 

Because  the  final  distribution  that  will  be  obtained  by  the  ques- 

* 

tioning  will  be  discrete,  the  real  world  events  must  be  viewed  as  hav¬ 
ing  some  rang<»  of  possible  values,  say  36,000  lb  +  1000  lb.  This  dis¬ 
tinction  will  not  enter  directly  into  the  use  of  the  probability 
distributions  that  are  obtained;  however,  it  is  related  to  the  engineer's 
ability  to  discriminate. 

The  questioning  proceeds  as  follows:  The  engineer  is  asked  to 

choose  between  the  gambles  shown  in  Table  6.  If  he  chooses  the  real 
world  gamble,  then  he  will  get  $10  if  the  thrust  turns  out  to  be 


All  techniques  described  in  this  Appendix  yield  discrete  proba¬ 
bility  distributions.  It  is  possible  to  obtain  continuous  subjective 
probability  distributions  but  the  techniques  require  that  the  subject 
be  able  to  evaluate  the  entire  distribution  at  one  time.  This  is  not 
as  easy  as  dealing  with  the  distribution  in  pieces,  and,  consequently, 
continuous  distributions  are  not  discussed.  Continuous  and  discrete 
distributions  are  described  in  Ref,  24. 

irk 

The  discussion  follows  Ref.  12,  pp.  256-258. 
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36,000  lb  (+  1000  lb)  and  $0  if  the  thrust  is  anything  else.*  If  he 
chooses  the  hypothetical  gamble,  he  will  get  $10  if  event  X  occurs,  and 

kk 

$0  if  event  Y  occurs. 

If  the  engineer  prefers  the  hypothetical  gamble,  it  is  inferred 
that  for  him  the  probability  that  the  engine  thrust  will  be  36,000  lb 
is  less  than  .7.  Hence,  he  is  offered  a  new  choice,  with  a  lower  prob¬ 
ability  of  event  X.  Let  the  new  choice  be  as  shown  in  Table  7.  Suppose 

Table  7 


A  SECOND  CHOICE-MAKING  SITUATION  TO  DETERMINE  THE  PROBABILITY 
DISTRIBUTION  FOR  THE  THRUST  OF  A  JET  ENGINE  UNDER  DEVELOPMENT 


that  the  engineer  is  indifferent  between  these  two  gambles.  Then  it  is 
inferred  that  his  subjective  probability  that  the  thrust  will  be  36,000  lb 
at  the  completion  of  the  project  is  .6  an  hat  his  probability  that 
the  thrust  will  be  anything  else  is  .4. 

Having  obtained  the  probability  for  the  thrust  being  36,000  lb, 
the  value  for  the  thrust  in  the  real  world  gamble  is  changed.  There 
may  be  some  difficulty  in  establishing  the  next  highest,  or  lowest, 
value  of  the  thrust  for  which  the  engineer  can  state  a  probability.  The 

The  $10  and  $0  consequences  are  used  so  the  engineer  will  be  able 
to  discriminate  on  a  basis  of  personal  value.  Any  two  consequences 
can  be  used  as  long  as  they  are  the  same  two  consequences  for  both  gam¬ 
bles.  It  is  not  necessary  that  the  engineer  actually  receive  the  con¬ 
sequences.  They  are  merely  part  of  a  construction  that  enables  the 
engineer  to  discriminate  and  evaluate. 

A  number  of  abstract  constructions  for  the  hypothetical  gamble 
can  be  used  to  facilitate  the  questioning.  For  example,  event  X  can 
be  thought  of  as  drawing  a  black  ball  from  a  bowl  containing  seven 
black  balls  and  three  white  balls.  Event  Y  is  then  the  drawing  of  a 
white  bail.  Similarly,  the  gamble  can  be  visualized  in  terms  of  a  lot¬ 
tery,  a  spinning  pointer,  or  a  ten-sided  die. 
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reason  for  this  possible  difficulty  is  that  the  engineer's  ability  to 
discriminate  between  values  of  the  thrust  is  limited.  Suppose  that  the 
engineer  attempts  to  determine  his  probability  for  the  thrust  being 
37,000  lb,  and  he  cannot,  or  he  gets  the  same  results  as  for  36,000  lb. 
This  indicates  that  the  engineer  cannot  discriminate  between  36,000  lb 
and  37,000  lb.  When  the  engineer  cannot  discriminate,  the  thrust  must 
be  changed  until  some  results  can  be  determined. 

The  end-points  of  the  distribution  are  determined  when  the  value 
of  the  thrust  reaches  points  for  which  the  engineer  is  indifferent  be¬ 
tween  the  gambles,  with  the  probability  of  event  X  being  0  and  the  prob¬ 
ability  of  event  Y  being  1.0.  Table  8  shows  some  hypothetical  results 
for  the  procedure  outlined.  The  end-points  are  at  34,000  lb  and 

Table  8 

PROBABILITY  DISTRIBUTION  FOR  THE  THRUST 
OF  AN  HYPOTHETICAL  JET  ENGINE  UNDER  DEVELOPMENT 


Thrust  (lb) 

Probability 

34,000 

0.0 

36,000 

0.6 

38,000 

0.2 

40,000 

0.1 

42,000 

0.1 

44,000 

0.0 

44,000  lb.  It  is  Important  to 

note  that  if  the  probability  of  44,CdO  ib 

is  0,  then  the  probability  of  anything  higher  will  also  be  0.  Thus, 
accurate  determination  of  the  end-points  requires  that  the  questioning 

procedure  begin  with  some  value 

of  the  thrust  that  lies  within  the  prob- 

ability  distribution  and  then  proceeds  in  an  orderly  fashion  to  both 
higher  and  lower  values  of  the  thrust. 

Note  also  that  the  probabilities  in  Table  8  satisfy  the  three  rules 
given  on  page  43  above.  If  the  conditions  specified  by  the  rules  are 
not  satisfied,  then  the  probabilities  must  be  normalized  or  redetermined. 

The  probabilities  shown  in  Table  8  reflect  the  engineer's  beliefs 
at  a  particular  time.  At  some  later  time,  his  probabilities  will  change 
because  of  the  additional  information  gained  in  the  interim. 
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Determination  of  the  Distribution  Function 
Using  Choice  between  Bets 

The  definition  of  the  real  world  gamble  in  Tables  6  and  7  may  not 
be  in  a  form  that  is  easily  comprehended  by  the  engineer.  The  form  re¬ 
quires  him  to  think  in  terms  of  a  probability  density  function,  and  it 
may  be  easier  for  him  to  think  in  terms  of  distribution  functions  or, 
more  precisely,  in  terms  of  the  areas  under  certain  portions  of  the 
density  functions. 

The  simplest  type  of  technique  for  determining  distribution  func¬ 
tions  uses  a  series  of  choices  of  the  form  shown  in  Fig.  11.  There  are 
only  two  events,  E^  and  and  two  bets  to  choose  between.  Each  event 
has  a  probability  of  .5.  This  situation  is  easy  for  an  individual  tc 
comprehend,  and  it  is  probably  the  most  common.  (More  people  have  en¬ 
countered  coin-tossing  than  any  other  random  process  having  "known" 
probabilities . ) 

The  procedure  involves  varying  the  value  of  T^  until  the  engineer 
is  indifferent  between  b^  and  b^ .  Then  it  is  inferred  that  the  proba¬ 
bility  that  the  thrust  will  be  less  than  T^  is  .5  and  the  probability 
that  the  thrust  will  be  greater  than  or  equal  to  T^  is  .5.  Thus,  T^ 
divides  the  probability  density  function  in  half. 

After  establishing  T^,  the  questioning  moves  down  one  level  as 
shown  in  Fig.  10.  At  this  level,  the  two  halves  of  the  probability  den¬ 
sity  function  are  halved.  The  values  T£,  T^,  and  divide  the  proba¬ 
bility  density  function  into  fourths;  that  is,  the  probability  that  the 
thrust  will  be  less  than  1 ^  is  one-fourth,  the  probability  that  it  will 
be  greater  than  or  equal  to  T£  and  less  than  is  one-fourth,  etc. 

The  process  is  continued  until  the  engineer  can  go  no  further.  As  an 
illustration,  suppose  that  the  engineer  is  successful  in  responding  at 
the  third  level  but  no  lower.  Some  hypothetical  results  are  shown  in 
Table  9,  and  the  corresponding  probability  distribution  function  in 
Table  10.  Two  adjustments  must  be  made  before  the  distribution  function 
can  be  used  for  sampling.  First,  the  "ends"  of  the  distribution  must 
be  determined.  Second,  the  values  of  thrust  to  be  used  in  the  sampling 
must  be  determined. 


II  -  Series  of  Choices  Between  Bets  to  Determine  a  Probability  Distribution  Function 


Table  10 


PROBABILITY  DISTRIBUTION  FUNCTION 
CORRESPONDING  TO  RESULTS  SHOWN 
IN  TABLE  9 

T.  p(t  ^ 


35.500 
36,000 
37,000 
38,000 

39.500 
40,000 

40.500 


.125 

.250 

.375 

.500 

.625 

.750 

.875 


The  highest  and  lowest  values  of  the  thrust  shown  in  Tables  9  and 
10  do  not  determine  the  end-points  of  the  distribution.  The  lowest 
value  is  35,000  lb.  The  probability  that  the  thrust  will  be  lower  than 
this  value  is  0.125.  However,  there  is  no  indication  of  how  much  lower 
the  thrust  may  be.  The  same  is  true  for  the  other  end  of  the  distribu¬ 
tion.  To  determine  the  ends,  the  procedure  described  in  the  previous 
section  is  used.  The  lower  end-point,  ,  satisfies  the  condition  that 
it  is  the  largest  value  of  the  thrust  for  which  p(t>T  )  =  1.0.  The 
upper  end-point,  T  ,  satisfies  the  condition  that  it  is  the  smallest 
value  for  which  p(t^T)=1.0  Table  11  shows  the  distribution  func- 
tion  with  end-points. 


Table  11 


PROBABILITY  DISTRIBUTION  FUNCTION 
OF  TABLE  10  WITH  END-POINTS 


T 

35,000 

35.500 
36,000 
37,000 
38,000 

39.500 
40,000 

40.500 
41,000 


P(t  <  T) 
.000 
.125 
.250 
.375 
.500 
.625 
.750 
.875 
1.000 


Some  adjustments  must  be  made  in  the  distribution  function  shown 
in  Table  11  to  put  it  into  a  suitable  form  for  sampling.  The  probabil¬ 
ity  that  the  thrust  will  be  between  35,000  lb  and  35,500  lb  is  0.125. 
For  the  purpose  of  calculation,  either  35,000  or  35,500  or  some  number 
in  between  must  be  selected.  The  simplest  choice  is  to  select  the  mid¬ 
point,  35,250.  This  is  shown  by  a  dashed  line  in  Fig.  12.  The  mid¬ 
points  for  all  other  values  are  also  indicated  by  dashed  lines. 
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T( lb  x  10-3  ) 

Fig.  12  -  Distribution  Function  for  Table  II 
The  Standard  Lottery 

The  standard  lottery  is  a  technique  for  determining  the  probabil- 
* 

ity  density  function. 

Suppose  that  [an  engineer  is  ]  offered  a  free  chance  at  a 
prize  of  value  V  under  the  following  conditions.  Balls  num¬ 
bered  1  to  100  have  been  placed  in  an  urn  and  one  of  these 
balls  has  then  been  drawn  and  put  in  a  closed  box.  [He  is] 
presented  with  100  tickets  numbered  from  1  to  100  and  [is] 
allowed  to  choose  one  of  them.  If  the  number  [he  chooses] 
matches  the  number  on  the  ball  which  has  been  drawn  from  the 
urn,  [he]  will  receive  the  prize;  if  not,  [he  receives]  noth¬ 
ing.  Suppose  further  that  even  though  the  prize  is  one 
which  [he  is]  extremely  anxious  to  win,  [he  does]  not  feel 
that  it  is  worth  the  slightest  effort  to  look  for  a  ticket 
with  any  particular  number  on  it;  [he]  simply  [takes]  the 
first  one  which  comes  to  hand. 

In  such  a  situation  [it  is  said  that  in  his ]  opinion  the 
100  possible  events  are  equally  likely.  Notice  very  careful¬ 
ly  that  [it  is  not  possible  to]  "prove"  that  the  events  are 
"in  fact"  equally  likely:  the  fact  is  that  the  ball  which  has 
been  drawn  has  some  one  particular  number  and  no  other.  But 


The  description  in  this  section  is  excerpted  from  Schlaifer  (see 
Ref.  29,  pp.  13-14).  Minor  changes  have  been  made  to  adapt  his  discus 
sion  to  the  context  of  the  jet  engine  development  used  to  illustrate 
the  preceding  techniques.  To  facil  jate  this  conversion,  the  events 
he  refers  to  have  been  replaced  by  the  values  of  the  thrust  shown  in 
Table  8.  Schlaifer  uses  the  word  "prize"  to  denote  the  same  thing  as 
"consequence . " 
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Aven  though  anyone  who  know  which  ball  has  boon  drawn  would 
not  bo  indifferent  among  the  100  tickets,  I  the  engineer's] 
decisions  must  b«  based  on  what  I hn  knows  or  believes]  about 
the  facta  of  the  world--they  cannot  be  based  on  the  unknown 
truth  about  these  facts.  Therefore  if  [tw  is  |  inditferent  in 
the  way  described,  then  f_or  I  h ini  |  the  100  events  are  equally 
likely  by  definition. 

Now  if  [the  engineer's]  state  of  mind  as  just  described 
is  to  be  described  by  numerical  weights  attached  to  tne  100 
possible  events  1,  2,  . ..,  100,  it  is  clear  that  these  weights 
must  all  be  equal.  If  the  sum  of  these  100  equal  numbers  is 
to  be  1,  as  required  by  Rule  1,  it  is  also  clear  that  the  num¬ 
ber  attached  to  each  event  must  be  1/100.  Rule  3  then  tells 
uo  that  events  such  as  "ball  number  2  or  7"  must  have  weight 
1/100  +  1/100  ■  2/100,  that  events  such  as  "any  ball  numbered 
between  1  and  37  inclusive"  must  have  weight  37/100,  and  so 
forth.  Thus  while  Rule  2  specified  only  that  the  weight  at¬ 
tached  to  any  event  must  be  a  number  between  0  and  1  inclu¬ 
sive,  [the  standard  lottery  describes]  a  way  of  selecting  a 
specific  number  within  this  range  to  describe  [a  person's] 
attitude  toward  any  conceivable  event  in  this  lottery. 

What  is  more  important,  [an  engineer]  can  find  the  unique 
set  of  weights  which  describes  his  attitudes  in  a  more  cn 
plex  situation  by  using  a  lottery  of  this  sort  as  a  standard 
of  comparison.  In  order  to  decide  what  weight  to  assign  to 
the  event  ("thrust  -  36,000  lb,"  in  Table  8],  thetengineer] 
can  Imagine  that  he  is  given  a  choice  between  a  certain  num¬ 
ber  of  tickets  in  the  standard  lottery  with  a  prize  of  value  V 
as  described  above  and  the  right  to  receive  this  same  prize 
to  the  event  ["thrust  ”  36,000  lb"].  If  in  his  opinion  the 
right  to  receive  this  prize  in  the  event  of  ["thrust  *  36,000 
lb"]  has  exactly  the  same  value  as  [60]  tickets  in  the 
standard  lottery,  then  by  definition  he  considers  these  two 
events  equally  likely  and  he  should  assign  weight  [60/100] 
to  the  event  ["thrust  =  36,000  lb"].  (It  goes  without  say¬ 
ing  that  if  the  standard  lottery  with  100  balls  does  not  of¬ 
fer  a  fine  encugh  division,  the  [engineer]  can  substitute  a 
similar  lottery  with  more  balls.  If  he  feels  that  the  right 
to  receive  the  prize  in  case  of  ["thrust  =  36,000  lb"]  is 
worth  more  than  [60]  tickets  but  less  than  [61]  in  a  lottery 
with  100  equally  likely  events,  he  may  decide  that  it  is 
equivalent  to  [605]  tickets  in  a  lottery  with  1000  equally 
likely  events.) 

Having  assigned  a  weight  to  the  event  ["thrust  =  36,000  lb"] 
the  [engineer]  can  proceed  in  the  same  way  to  assign  weights 
to  all  the  other  events  in  Table  [8]. These  weights  must  of 
course  be  such  that  their  total  is  1,  and  therefore  what  the 
[engineer]  is  really  doing  is  placing  the  set  of  collectively 
exhaustive  and  mutually  exclusive  events  shown  in  Table  [8] 
into  one-to-one  correspondence  with  a  set  of  collectively  ex¬ 
haustive  and  mutually  exclusive  events  in  the  standard  lot¬ 
tery.  When  he  is  through,  the  event  ["thrust  =  36,000  ib"] 
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will  correspond,  say,  to  the  event  "ball  numbered  between  1 
and  [60]  inclusive,"  the  event  ["thrust  =  38,000  lb"]  to  balls 
[61  to  80],  and  so  forth.  It  is  perhaps  worth  remarking 
that  [it  is]  in  no  sense  [assumed]  that  [an  engineer]  will 
actually  be  as  ready  to  gamble  on  balls  drawn  from  an  urn  as 
to  make  decisions  concerning  his  regular  business.  We  are 
simply  assuming  that  a  rational  person  can  with  practice 
think  abstractly  about  his  feelings  of  certainty  and  uncer¬ 
tainty  in  any  given  situation,  regardless  of  any  feelings  he 
may  have  about  any  other  aspects  of  the  situation. 


Determination  of  Multivariate  Probabilities 

When  subjective  probabilities  are  required  for  a  number  of  vari- 

•ft 

ables  that  are  correlated,  the  information  must  be  gathered  as  a  mul¬ 
tivariate,  or  joint,  probability  distribution.  A  multivariate  dis- 
tribution  is  a  probability  distribution  over  the  possible  outcomes 
of  several  variables.  In  the  case  of  two  variables,  and  ,  the  dis¬ 
tribution  is  of  the  form  p(x^,  X2),  where  x^  and  X£  are  specific  val¬ 
ues  of  the  variables  and  X2 ,  respectively. 

There  appear  to  be  no  papers  published  regarding  the  determination 
of  multivariate,  subjective  probabilities.  However,  the  techniques  de- 
'•cri.tcd  above  ccn  be  extended  to  the.  bank.  Consider,  for  example,  the 
case  of  determining  a  bivariate  distribution.  The  standard  lottery  can 
be  used  by  having  the  subject  distribute  the  tickets  over  an  array  of 
pairs  of  values  for  the  two  variables.  The  choice  between  gambles  and 
the  choice  between  bets  techniques  can  be  applied  by  fixing  the  value 
of  one  variable  and  running  through  the  questioning  procedure  for  pos¬ 
sible  values  of  the  other  variable.  Then  the  value  of  the  first  vari¬ 
able  is  changed  again,  and  so  on,  until  the  distribution  is  determined. 

A  difficulty  arises  because,  as  the  number  of  variables  increases, 
the  subject's  ability  to  give  meaningful  responses  becomes  increasingly 
impaired.  This  is  due  to  the  increase  in  the  number  of  dimensions  that 
must  be  constantly  kept  in  mind.  Hence,  some  experimentation  is  re¬ 
quired  before  the  usefulness  of  multivariate  subjective  probabilities 
can  be  determined. 

This  i s  the  case  as  described  in  the  text  under  "Correlation 
among  Inputs,"  p.  20. 

See  Kef.  24,  pp.  285-292,  for  a  more  thorough  discussion  of  mul¬ 
tivariate  (joint)  probabilities.  i ' 

I 

I ) 

j  ; 


I 


1 

i 
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SELECTION  OF  PROBABILITY  APPRAISERS 

Use  of  subjective  probabilities  as  guides  tor  decisionmaking  de¬ 
mands  careful  selection  of  probability  appraisers.  The  main  concern 

is  the  "quality"  of  the  responses.  Several,  studies  have  been  made  of 

* 

people  s  ability  to  estimate  the  probabilities  of  random  events. 

Such  events  have  objective  probabilities,  and  in  these  studies  it  was 
possible  to  compare  the  subjective  probabilities  and  the  "true"  prob- 
abilities.  However,  when  the  events  being  considered  are  not  the  re¬ 
sults  of  random  processes,  it  is  not  possible  to  verify  the  accuracy 
of  the  responses  because  there  are  no  "true"  probabilities.  The  impos¬ 
sibility  of  evaluating  subjective  probabilities  for  non-random  events 
forces  attention  to  evaluating  the  ability  of  people  to  give  "meaning¬ 
ful"  responses. 

The  evaluation  and  selection  of  probability  appraisers  is  a  quali¬ 
tative  subject.  At  present  there  are  no  quantitative  tests  of  an  in- 
dividual's  ability  to  express  "good"  subjective  probabilities.  As 

mentioned  above,  it  may  be  desirable  to  conduct  tests  such  as  those 
reported  in  Ref.  20,  pp.  321-327.  However,  there  are  many  other  factors 
which  have  a  bearing  on  an  individual's  ability  as  a  probability  ap¬ 
praiser.  The  best  that  can  be  done  here  is  to  mention  them. 

The  factors  which  have  some  bearing  on  an  individual's  ability  to 
give  "meaningful"  probability  appraisals  can  be  classified  as  being 
either  personal  or  environmental.  Personal  factors  include  intelligence, 
knowledge,  and  factors  that  relate  to  the  possibility  of  bias,  both  inten¬ 
tional  and  unintentional.  For  example,  a  person  of  high  integrity  will 
be  less  likely  to  bias  his  responses  than  a  person  of  low  integrity. 


Ref.  20,  pp.  321-327. 

Most  of  the  studies  indicated  that  people  overestimate  low  prob¬ 
abilities  (less  than  approximately  .20),  and  underestimate  high  prob¬ 
abilities  (greater  than  approximately  .20).  This  phenomenon  may  need 
to  be  compensated  for  when  subjective  probabilities  are  used  in 
decisionmaking. 

JUUL 

rfTv/f 

A  scheme  has  been  proposed  by  De  Finetti  (see  Ref.  8)  for  eval¬ 
uating  individual  probability  appraising  skills  on  the  basis  of  pa 
performance. 
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Unintentiona 1  bias  may  be  introduced  by  pessimism  or  optimism.  Measure¬ 
ment  of  these  factors  is  difficult  if  not  impossible.  Until  adequate 
psychological  tests  can  be  developed  (if  they  ever  can)r  personal, 
common-sense  appraisals  of  individuals,  using  records  of  past  perform¬ 
ance  and  other  personal  factors,  will  have  to  suffice. 

Environmental  factors  include  influences  that  are  external  to  the 
individual.  The  amount  of  bias  resulting  from  these  influences  depends 
on  the  interaction  between  the  personal  factors  and  the  environmental 
factors.  Examples  of  environmental  factors  are:  specific  requirements 
given  in  the  contract,  company  aspirations,  the  management  position  of 
the  individual,  and  the  punishment  and  reward  structure.  Contract  re¬ 
quirements  and  company  aspirations  may  influence  an  individual  to  bias 
his  responses  to  please  his  superiors.  An  individual's  position  in  the 
management  structure,  and  the  punishment  and  reward  structure  interact 
with  the  individual's  personal  values  and  integrity.  This  is  probably 
what  is  meant  by  an  individual's  "degree  of  involvement." 

In  spite  of  the  numerous  adverse  influences  that  can  introduce 
bias  in  the  case  of  a  responsible  engineer,  in  most  cases  he  will  be 
the  best  source  of  information.  What  must  be  done  is  to  develop  tech¬ 
niques  that  increase  the  objectivity  of  the  responses. 

Discussions  of  methods  for  increasing  the  objectivity  of  subjec¬ 
tive  probabilities  have  concentrated  on  the  elimination  of  bias.  The 

two  approaches  that  have  received  the  most  attention  are  rewarding 
* 

"good'  performance,  and  comparing  and  combining  the  probabilities  ob- 
tained  from  two  or  more  individuals.  Another  device  that  may  lead 
to  improved  results  is  to  have  the  probability  appraiser  prepare  a  list 
of  factors  that  he  considered  when  he  determined  his  probabilities. 

Such  a  list  could  be  of  value  when  comparing  probabilities  obtained 
from  different  people  and  probabilities  obtained  from  a  single  person 
at  different  times. 

Ref.  12 ,  p .  261. 

Ref.  12,  p.  262;  Ref.  9;  and  Ref.  13,  pp.  16-18. 
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Appendix  C 

GENERATING  PROBABILITY  DISTRIBUTIONS 
USING  MONTE  CARLO  SIMULATION 


This  Appendix  describes  th<_  process  of  generating  a  probability 
distribution  using  Monte  Carlo  simulation.  The  example  used  is  that 
of  generating  a  probability  distribution  for  the  life  of  a  two -component 
electronic  device,  based  on  the  known  probability  distributions  for  the 
operating  lives  of  the  components  (tubes),  and  the  design  equation  which 
reflects  the  behavior  of  the  device  in  terms  of  the  behavior  of  its 
components.  The  "design  equation"  in  this  case  states  that  the  device 
fails  if  either  one  of  its  components  fails.  An  example  more  closely 
related  to  the  aircraft  project  used  throughout  this  Memorandum  is  that 
of  determining  the  probability  distribution  for  the  possible  values  of 
the  aspect  ratio  of  an  aircraft,  given  the  probability  distributions 
for  the  wing  span  and  the  wing  area.  The  example  of  an  electronic  de¬ 
vice  was  chosen  because  the  mathematical  calculations  required  are 
simpler.  However,  the  equivalence  of  these  two  examples  is  demonstrated 
by  noting  that  they  both  are  of  the  form:  Given  p^x^),  p2(x2>,  and 
y  =  f(x^,  x^),  determine  p(y). 

* 

The  electronic  device  example  requires  a  comparison  of  only  two 
numbers  and  selection  of  the  smaller,  whereas  the  aspect  ratio  example 
requires  the  squaring  of  one  number  and  division  of  that  by  a  second 
number.  In  an  operational  situation,  the  simulation  would  be  performed 
by  computer,  using  available  Monte  Carlo  programs,  in  which  case  the 
more  extensive  calculations  would  be  accomplished  with  a  minimum  of  effort. 
** 

In  the  electronic  device  example  p^(x^)  and  p2 (x7 )  arc  the  prob¬ 
ability  distributions  of  the  operating  lives  of  the  components;  y  is  the 
the  life  of  the  device;  y  =  fCxj,  x2)  =  min  (xj,  x2>;  and  p(y)  is  the 
probability  distribution  of  the  operating  life  of  the  device.  in  the 
aspect  ratio  example,  PjO^)  and  P2(x2>  are  the  probability  distributions 

for  the  wing  span  and  the  wing  area,  respectively;  y  is  the  aspect  ratio; 

2 

y  =  f ( x | ,  x2)  =  x j /x2 ;  and  p(y)  is  the  probability  distribution  over  the 

possible  values  of  the  aspect  ratio.  This  formulation  can  be  extended 

to  any  number  of  characteristics  as  follows:  Let  x, ,  ....  x  denote 

I  n 
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Suppose  that  the  life  characteristics  of  two  electronic  tubes  are 
as  shown  in  Figs.  13  and  14.  To  use  the  Monte  Carlo  technique,  these 

i 

curves  must  be  changed  from  density  functions  to  distribution  functions. 
This  is  done  in  Figs.  15  and  16.  Values  along  the  probability  scale 
of  the  distribution  function  are  selected  by  means  of  a  table  of  ran¬ 
dom  numbers.  In  the  case  shown,  the  digits  0  to  9  can  be  used.  (In 
cases  where  the  probabilities  involve  two  digits,  pairs  of  digits  must 
be  selected  from  a  random  number  table.)  When  a  random  number  is  se¬ 
lected,  a  horizontal  line  is  drawn  from  the  Y-axis  on  the  cumulative 
probability  curve  until  it  hits  one  of  the  vertical  lines.  This  de¬ 
termines  a  value  for  the  life  of  the  tube.  For  example,  if  the  random 
number  is  8,  then  the  life  of  Tube  No.  1  is  260  hr.  To  facilitate 
working  out  this  example,  the  data  in  Figs.  15  and  16  are  converted  in¬ 
to  tabular  form  in  Tables  12  and  13. 

To  determine  a  sample  lifetime  for  the  electronic  device,  a  sample 
lifetime  is  determined  for  each  tube  and  the  shortest  is  the  lifetime 
for  the  device.  Suppose  the  sample  lifetime  for  Tube  No.  1  is  260  hr, 

the  component  characteristics  of  some  system  and  y  ,  y  denote  the 

1  m 

system  performance  characteristics.  The  problem  is  then:  Given  the 
probability  distributions  p^(x^),  ...»  p^Cx^),  flnd  the  design  equations 


1  *lv'vl’  An'  ’  ym  “  •••>  xn)  >  determine  the  proba¬ 

bility  distributions  Pn+1<y1),  ....  Pn+m(ym>-  The  aircraft  example  in 
Sec.  II  can  be  stated  in  these  terms  by  making  the  following  identifi¬ 
cations  : 


X1  V  x2 


J  ^  |  X 

L  J  0 


4  ~  *  X5  k,  xA  S,  x7  e. 


X8  =  T>  x9  =  x10  =  C’’  y 


1  vmax’  y2  "  Pac’  '3  ‘max’  '4  max 
Then  the  Pj(x^)  are  given  in  Table  2;  the  design  equations  are  Eqs.  (1), 
(3),  (6),  and  (9);  and  the  Pj(Yj)  are  given  in  Table  3. 


=  R 


6  ~7 

.»  y,,  =  E 


See  Ref.  5,  pp.  179-183,  for  further  discussion.  The  distribu¬ 
tion  function  shows  the  cumulative  probability.  The  term  cumulative 
probability  is  used  to  indicate  a  particular  value  of  the  distribution 
function. 


Cumulative  probability  of  failure  Probability  of  failure 
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Fig.  13  -  Life  Curve 
for  Tube  No.  I 


Fig.  14  -  Life  Curve 
for  Tube  No.  2 
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Fig.  15  -  Cumulative  Probability 
of  Failure  for  Tube  No.  I 
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Fig.  16  -  Cumulative  Probability 
of  Failure  for  Tube  No.  2 
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Table  12 

SAMPLING  DISTRIBUTION  FOR  TUBE  NO.  1 


Random  Number 

Corresponding  Life 
of  Tube  No.  1 

9 

270  hr 

8-7 

260 

6-3 

250 

2-1 

240 

0 

230 

Table  13 


SAMPLING  DISTRIBUTION  FOR  TUBE  NO.  2 


Random  Number 

Corresponding  Life 
of  Tube  No.  2 

9-7 

260  hr 

6-4 

250 

3-2 

240 

1 

230 

0 

220 

as  determined  above.  Another  random  number  is  drawn  for  Tube  No.  2. 

Suppose  it  is  3;  the  lifetime  of  Tube  No.  2  is  then  240  hr.  In  this 

instance,  the  lifetime  of  the  device  is  240  hr. 

To  determine  the  distribution  of  lifetimes  for  the  device,  the 

above  process  is  repeated  a  large  number  of  times.  The  frequencies  of 

the  observed  lifetimes  of  the  device  are  plotted  as  a  bar  chart.  This 

results  in  a  life  curve  for  the  device.  The  number  of  times  that  this 

process  is  repeated  depends  on  the  desired  accuracy.  The  larger  the 

number,  the  more  accurate  the  resulting  life  curve  will  be.  Accuracy 

requirements  vary  among  applications.  In  all  cases,  the  sample  site 

is  determined  using  standard  statistical  techniques  and  will  not  be 

* 

discussed  in  this  Memorandum. 

For  illustrative  purposes,  the  life  curve  of  the  two-tube  elec¬ 
tronic  device  is  determined  by  obtaining  a  sample  of  25  pairs  of  life¬ 
times  for  the  two  tubes.  First,  a  table  of  the  form  of  Table  14  is  set  up. 

*See  Ref.  29,  pp.  157-159,  275-277. 
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Table  14 


TWENTY-FIVE  SAMPLES  OF  THE  LIFETIME  FOR  A  TWO-TUBE  ELECTRONIC  DEVICE 


j 


Tube  No .  1 

Tube  No.  2 

Device 

Lifetime 

Random 

Number 

Lifetime 
(from  Table  12) 

Random 

Number 

Lifetime 
(from  Table  13) 

0 

230  hr 

9 

260  hr 

230  hr 

5 

250 

4 

250 

250 

4 

250 

2 

240 

240 

0 

230 

1 

230 

230 

8 

260 

0 

220 

220 

0 

230 

6 

250 

230 

0 

230 

6 

250 

230 

2 

240 

6 

250 

240 

5 

250 

7 

260 

250 

7 

260 

9 

260 
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Random  numbers  taken  from  Table  7-3  of  Ref.  5. 


Then  random  numbers,  25  for  each  tube,  are  obtained  from  Tables  12  and 
13.  The  lifetime  of  the  device  is  the  lifetime  of  the  first  tube  to 
fail.  A  table  of  the  frequencies  of  the  various  lifetimes  of  the  de¬ 
vice  is  constructed  (Table  15). 


Table  15 

LIFE  CURVE  DERIVED  FROM  TABLE  14 


Device  Lifetime 

Frequency 

%  or  Probability 

220 

4 

.  16 

230 

5 

.20 

240 

5 

.20 

250 

9 

.36 

260 

2 

.08 

25 

l  .00 

Probability  of  f a  i  1 urc 


These  results  are  plotted  as  probabilities  of  failure  (the  life 
curve)  in  Fig.  17,  and  as  cumulative  probabilities  of  failures  in  Fig.  18. 


Lifetime  (hr' 

Fig.  17  -  Life  Curve  for  Two-Tube 
Electronic  Device 


Lifetime  (hr' 

Fig.  18  -  Cumulative  Probability 
of  Failure  for  Two-Tube 
Electronic  Device 
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Appendix  D 
STATISTICAL  MEASURES 

Decisionmaking  under  uncertainty  is  usually  based  on  a  few  summary 
statistics  which  characterize  the  probability  distribution  of  the  pos¬ 
sible  results.  Those  most  widely  used  are  measures  of  central  tendency 
and  measures  of  dispersion.  This  Appendix  reviews  the  definitions  of 
these  measures  and  also  discusses  probabilities  related  to  specific 
levels  of  performance.  It  is  included  for  those  readers  who  may  wish 
to  briefly  review  the  meaning  and  definition  of  these  measures. 

MEASURES  OF  CENTRAL  TENDENCY 

Measures  of  central  tendency  indicate  where  the  middle  of  the 
distribution  lies.  The  mean  is  the  average,  or  expected,  value.  It 
is  given  by 

H  =  E(r)  =  Z  r  p(r  )  .  (10) 

i 


The  mode  is  the  most  likely  value  of  r;  that  is,  the  value  that  occurs 
most  frequently.  In  mathematical  terms  it  is  that  r^  such  that  p(r^)  - 
p(r )  for  all  i  /  j.  The  median  is  the  value  of  r  that  approximately 
divides  the  area  under  the  distribution  in  half.  Mathematically,  it 


is  that  r.  such  that  p(r 
J 


\  and  p(r  >  r^) 


where 


P(r  <  r  )  =  v^j'1  p(r.) 


and 


/  ^  \  -1=1  max*  /  \ 

P(r  >  t.)  -  v,i_.+1  p(r.) 


Measures  of  central  tendency  are  measured  in  the  same  units  as  the 
variable  in  question.  For  example,  if  r  in  Table  16  is  the  speed  of 


* 

If  the  distr ibution  is  continuous,  then  the  median  divides  the 
area  exactly  in  half.  See  Ref.  24,  p.  213. 
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an  aircraft  in  miles  per  hour,  then  the  mean  is  p,  =  485  miles  per  hour, 
the  mode  is  400  miles  per  hour,  and  the  median  is  500  miles  per  hour. 

MEASURES  OF  D1SPKRSION 

Measures  of  dispersion,  which  indicate  the  spread  about  the  central 
tendency,  are  especially  useful  in  measuring  uncertainty.  For  example, 
the  ultimate  outcome  from  a  distribution  with  high  dispersion  is  more 
uncertain  than  the  ultimate  outcome  from  a  distribution  with  low 
d 1 spersion . 


Table  16 

PROBABILITY  DISTRIBUTION  FOR 
ILLUSTRATION  OF  STATISTI'  AL  MEASURES 


r 

_ g-Cr)  _ 

Tj  *  200 

o 

o 

• 

r2  *  300 

.10 

*  400 

.  35 

r.  ■  500 

4 

.25 

r5  =  600 

o 

CSI 

• 

r,  -  700 

.10 

r?  =  800 

.00 

The  range ,  R,  is  the  difference  between  the  values  of  the  variable 
in  question  for  which  the  probabilities  are  not  zero.  In  mathematical 
terms,  the  range  is  given  by  r^  -  r^,  where  r^  >  r^ ,  p(ri>  >  0,  p(r^)  >  0, 
p(r[))  =  0  for  all  rh  r^,  and  p(rk>  =••  0  for  all  r^  >  r^.  The  unit  of 
measure  of  the  range  is  the  same  as  for  the  variable  in  question.  In 
Table  16,  R  ■  700  -  300  “  400  miles  per  hour. 

There  are  limitations  to  the  usefulness  of  the  range.  For  example, 
the  three  distributions  shown  in  Fig.  19  have  the  same  range  but  they 
very  different.  A  more  general  measure  of  dispersion  that  would 


are 
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dlstinguish  between  the  three  distributions  shown  in  Fig.  19  is  the 
inter-dth  percentile  range.  This  is  a  measure  of  the  central  range 
between  which  the  dth  percentage  of  the  distribution  lies.  In 
mathematical  terms  r^-r^  Is  the  dth  percentile  range  when 

p(r  <  r£)  s  y,  p(r  >  rf )  '  y,  and 


/  ^  \  ,  1-d  ,  .  ,  1+d 

p(r  >  ry)  £  y ,  p(r  <  ry)  <  y  . 


As  a  specific  example,  if  (l-d)/2  «=  .25  and  (l+d)/2  m  .75,  then  ru_r^ 

gives  the  interquartile  range.  For  the  example  in  Table  16,  r  = 

e 

400  miles  per  hour  and  r^  =  600  miles  per  hour;  hence,  the  inter¬ 
quartile  range  is  200  miles  per  hour. 


Fig,  19  -  Density  Functions  with  the 
Same  Means  and  Ranges  but 
Diffeient  Variances 


The  var  i  ance ,  r'  ,  of  a  distribution  is  a  measure  of  the  spread, 
or  dispersion,  about  the*  mean,  n.  For  discrete  distributions,  it  can 
be  calculated  using  the  following  expression: 

'2(r)  »  V  r2p(r  )  -  \iZ  . 
i  1  1 


(12) 
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The  unit  of  measure  of  the  variance  is  the  square  of  the  unit  of 

2 

measure  of  the  variable  in  question.  In  Table  16,  o  =  13,275  (miles 
per  hour) 

If  the  variance  is  small,  then  the  area  under  the  distribution 
is  concentrated  near  the  mean.  If  the  variance  is  large,  then  the 
area  is  widely  dispersed.  This  is  illustrated  in  Fig.  19,  where  the 
three  density  functions  A,  B,  and  C  have  the  same  mean  and  the  same 
range  but  increasing  variance.  The  distributions  shown  are  continuous 
(not  discrete)  to  simplify  the  illustration. 

The  standard  deviation,  o,  is  the  positive  square  root  of  the 
variance: 

o(r)  -  */cr2  (r)  •  (13) 

In  Table  16,  o  =  approximately  115  miles  per  hour. 

MEASURES  RELATED  TO  A  REQUIRED  LEVEL  OF  PERFORMANCE 

In  the  case  of  most  weapon  system  development  projects,  the  critical 
system  performance  characteristics  have  specified  minimum  performance 
levels  usually  referred  to  as  specifications.  In  these  cases,  the 
measures  of  central  tendency  and  dispersion  are  of  limited  value  be¬ 
cause  they  do  not  relate  directly  to  the  specifications.  For  example, 
consider  a  project  to  develop  an  aircraft  with  a  minimum  speed  (under 
certain  specified  conditions)  of  750  mph.  Assume  that  the  probabilistic 
information  described  above  has  been  obtained,  and  it  is  found  that  the 
expected  speed  is  700  mph  with  a  standard  deviation  of  75  mph.  Assuming 
the  distribution  to  be  normal,  this  means  that  there  is  a  probability 
of  0.68  that  the  speed  of  the  aircraft  will  be  between  625  mph  and  775 
mph.  It  would  be  more  relevant  to  have  a  figure  for  the  probability 
of  meeting  or  exceeding  the  required  performance  level;  that  is,  the 
probability  that  the  aircraft  will  fly  750  mph  or  more.  This  is  easily 
obtained  from  the  probability  distribution  over  possible  speeds.  It  is 
obtained  by  summing  the  probabilities  for  all  speeds  equal  to  the  spec- 
fied  speed  and  greater.  Let  r^  indicate  the  possible  speeds  of  the 
aircraft  and  the  specified  speed.  Then 
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r 

max 

P(r  2  r  )  =  £  P^)*  (L4) 

r  .=r 
1  s 

where  r  is  the  maximum  possible  speed  indicated  by  the  distribution. 

The  probability  of  meeting  or  exceeding  the  specification  is 
similar  to  measures  of  central  tendency  in  that  it  does  not  give  any 
indication  of  how  much  the  specification  may  be  exceeded;  that  is,  no 
measure  of  dispersion.  Therefore,  the  probability  of  meeting  or  ex¬ 
ceeding  the  specif 4  "lt' ? may  be  supplemented  by  a  mea.c"ro  such  as  the 
probability  of  exceeding  the  specifications  by  more  than  some  percentage, 
say  x.  This  is  given  by 


p(r  >  rg  +  xrg)  =  )  (15) 

r  =r  +xr 
i  s  s 

This  measure  is  especially  useful  in  situations  where  a  substantial 
overdesign  is  undesirable.  Continuing  the  previous  example,  suppose 
that  the  aircraft  under  development  would  not  be  desirable  if  its  speed 
turned  out  to  be  greater  than  1000  mph."  This  corresponds  to  exceeding 
the  specifications  by  33  percent;  that  is,  x  -  0.33.  If  the  probability 
of  exceeding  1000  mph  is  too  high,  then  some  changes  in  the  project 
may  be  made. 

This  measure  is  also  valuable  when  parallel  projects  are  compared 
or  when  progress  on  one  given  project  is  compared  with  data  that  has 
been  obtained  at  two  different  times  (see  Sec.  III).  In  such  cases, 
it  can  be  used  to  discriminate  between  two  situations  that  may  have  the 


The  decrease  of  desirability  as  speed  increases  may  be  due  to 
changes  in  other  performance  parameters,  such  as  an  increase  in  the 
minimum  turning  radius,  or  it  may  reflect  a  desire  to  save  time  and 
money  rather  than  achieve  a  higher  speed. 

Both  x  and  what  probability  is  "too  high"  must  be  determined 
by  the  user  of  the  information. 
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sai  ,■  probabilities  of  meeting  or  exceeding  the  specification  but  that 
have  different  probabilities  of  exceeding  the  spectif ication  by  x  percent. 

The  difference  between  the  probability  of  meeting  or  exceeding  the 
specification  and  the  probability  of  exceeding  the  specification  by 


more  than  x  percent  is  the 


probability  of  being  within  the  acceptable 


p(ls  +  xrs  >  r  rs>  =  p(r  “  rs>  “  P<r  >  rs  +  xrs)‘ 


(16) 
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